System and methods of universal parameterization of holographic sensory data generation, manipulation and transport

ABSTRACT

A method determines four dimensional (4D) plenoptic coordinates for content data by receiving content data; determining locations of data points with respect to a first surface to creating a digital volumetric representation of the content data, the first surface being a reference surface; determining 4D plenoptic coordinates of the data points at a second surface by tracing the locations the data points in the volumetric representation to the second surface where a 4D function is applied; and determining energy source location values for 4D plenoptic coordinates that have a first point of convergence.

TECHNICAL FIELD

This disclosure is generally related to generation of holographiccontent comprising sensory information, and more specifically togeneration of holographic content from non-holographic information.

BACKGROUND

The dream of an interactive virtual world within a “holodeck” chamber aspopularized by Gene Roddenberry's Star Trek and originally envisioned byauthor Alexander Moszkowski in the early 1900s has been the inspirationfor science fiction and technological innovation for nearly a century.However, no compelling implementation of this experience exists outsideof literature, media, and the collective imagination of children andadults alike.

SUMMARY

In an embodiment, a method for determining four dimensional (4D)plenoptic coordinates for content data may comprise receiving contentdata; determining locations of data points with respect to a firstsurface to create a digital volumetric representation of the contentdata, the first surface being a reference surface; determining 4Dplenoptic coordinates of the data points at a second surface by tracingthe locations the data points in the volumetric representation to thesecond surface where a 4D function is applied; and determining energysource location values for 4D plenoptic coordinates that have a firstpoint of convergence.

In an embodiment, a method for determining four dimensional (4D)plenoptic coordinates for content data may comprise receiving contentdata; determining locations of data points with respect to a referencepoint location; vectorizing the data point by creating vectors of thedata points based on the reference point location; determining, based onthe vectorized data points, locations of data points with respect to afirst surface to creating a digital volumetric representation of thecontent data, the first surface being a reference surface; anddetermining 4D plenoptic coordinates of the data points at a secondsurface by tracing locations the data points in the volumetricrepresentation to the second surface where a 4D function is applied.

In an embodiment, a method of vectorization may comprise receiving firstcontent data; identifying a surface in the content data; determining asurface identification of the surface; determining material propertydata of the surface; associating the surface identification with thematerial property data of the surface; creating the vectors of thematerial property data; and generating vectorized material property databased on the created vectors.

In an embodiment, a system for determining four dimensional (4D)plenoptic coordinates for content data may comprise an input-outputinterface operable to receive content data; a processing subsystem incommunication with the input-output interface and comprising a sensorydata processor, a vectorization engine, and a tracing engine; whereinthe sensory data processor is operable to determine locations of datapoints within the content data, with respect to a first surface, and tocreate a digital volumetric representation of the content data, thefirst surface being a reference surface; wherein the tracing engine isoperable to determine 4D plenoptic coordinates of the data points at asecond surface by tracing the locations of the data points in thedigital volumetric representation to the second surface where a 4Dfunction is applied; and wherein the tracing engine is operable todetermine energy source location values for the 4D plenoptic coordinatesthat have a first point of convergence.

In an embodiment, a system for determining four dimensional (4D)plenoptic coordinates for content data may comprise an input-outputinterface operable to receive content data; a processing subsystem incommunication with the input-output interface and comprising a sensorydata processor, a vectorization engine, and a tracing engine; whereinthe sensory data processor is operable to determine locations of datapoints within the content data with respect to a reference pointlocation; wherein the vectorization engine is operable to vectorize thedata points based on the reference point location; wherein the sensorydata processor is further operable to determine, based on the vectorizeddata points, locations of data points with respect to a first surface tocreate a digital volumetric representation of the content data, thefirst surface being a reference surface; and wherein the tracing engineis operable to determine 4D plenoptic coordinates of the data points ata second surface by tracing locations of the data points in thevolumetric representation to the second surface where a 4D function isapplied.

In an embodiment, a system for vectorization may comprise aninput-output interface operable to receive content data; a processingsubsystem in communication with the input-output interface andcomprising a vectorization engine; wherein the vectorization engine isoperable to identify a surface in within the content data, to determinea surface identification of the surface, to determine a materialproperty data of the surface, and to associate the surfaceidentifications with the material property data of the surface; andwherein the vectorization engine is further operable to create vectorsof the material property data, and to generate vectorized materialproperty data based on the created vectors.

In an embodiment, a system for determining four dimensional (4D)plenoptic coordinates for content data may comprise an input-outputinterface operable to receive content data; a processing subsystem incommunication with the input-output interface and comprising a sensorydata processor, a vectorization engine, and a tracing engine; acompression engine in communication with the processing subsystem andthe input-output interface; and an optional memory in communication withthe compression module, the input-output interface, and the processingsubsystem; wherein the sensory data processor is operable to determinelocations of data points within the content data with respect to areference point location; wherein the vectorization engine is operableto vectorize the data points based on the reference point location;wherein the sensory data processor is further operable to determine,based on the vectorized data points, locations of data points withrespect to a first surface to create a digital volumetric representationof the content data, the first surface being a reference surface;wherein the tracing engine is operable to determine 4D plenopticcoordinates of the data points at a second surface by tracing locationsof the data points in the volumetric representation to the secondsurface where a 4D function is applied; wherein the compression engineis operable to receive data from the processing subsystem, to compressthe data, and to either store the compressed data in the optional memoryor send the compressed data to the input-output interface; and whereinthe optional memory is configured to receive data from the input-outputinterface, the processing subsystem, and the compressing engine, tostore the data, and to send the stored data to either the input-outputinterface, the processing subsystem, or the compression engine.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating design parameters for anenergy directing system;

FIG. 2 is a schematic diagram illustrating an energy system having anactive device area with a mechanical envelope;

FIG. 3 is a schematic diagram illustrating an energy relay system;

FIG. 4 is a schematic diagram illustrating an embodiment of energy relayelements adhered together and fastened to a base structure;

FIG. 5A is a schematic diagram illustrating an example of a relayedimage through multi-core optical fibers;

FIG. 5B is a schematic diagram illustrating an example of a relayedimage through an energy relay that exhibits the properties of theTransverse Anderson Localization principle;

FIG. 6 is a schematic diagram showing rays propagated from an energysurface to a viewer;

FIG. 7A illustrates a perspective view of an energy waveguide systemhaving a base structure, four energy devices, and four energy relayelements forming a seamless energy surface, in accordance with oneembodiment of the present disclosure;

FIG. 7B illustrates an energy relay system according to one embodimentof the present disclosure;

FIG. 7C illustrates a top-down perspective view of an embodiment of anenergy waveguide system according to one embodiment of the presentdisclosure;

FIG. 7D illustrates a front perspective view of the embodiment shown inFIG. 7C;

FIGS. 7E-7L illustrate various embodiments of an energy inhibitingelement;

FIG. 8 is a flow chart illustrating an embodiment of an embodiment of aprocess for processing holographic sensory data;

FIG. 9 is a schematic diagram of a virtual environment constructed fromsensory data;

FIG. 10 is a schematic diagram illustrating an embodiment of energytracing;

FIG. 11 is a schematic diagram illustrating an embodiment of an energydirecting device 1000 going through a tracing process;

FIG. 12 is a schematic diagram of a processing system for processholographic sensory data; and

FIG. 13 is a block diagram illustrating an embodiment of a vectorizationprocess

DETAILED DESCRIPTION

An embodiment of a Holodeck (collectively called “Holodeck DesignParameters”) provide sufficient energy stimulus to fool the humansensory receptors into believing that received energy impulses within avirtual, social and interactive environment are real, providing: 1)binocular disparity without external accessories, head-mounted eyewear,or other peripherals; 2) accurate motion parallax, occlusion and opacitythroughout a viewing volume simultaneously for any number of viewers; 3)visual focus through synchronous convergence, accommodation and miosisof the eye for all perceived rays of light; and 4) converging energywave propagation of sufficient density and resolution to exceed thehuman sensory “resolution” for vision, hearing, touch, taste, smell,and/or balance.

Based upon conventional technology to date, we are decades, if notcenturies away from a technology capable of providing for all receptivefields in a compelling way as suggested by the Holodeck DesignParameters including the visual, auditory, somatosensory, gustatory,olfactory, and vestibular systems.

In this disclosure, the terms light field and holographic may be usedinterchangeably to define the energy propagation for stimulation of anysensory receptor response. While initial disclosures may refer toexamples of energy and mechanical energy propagation through energysurfaces for holographic imagery and volumetric haptics, all forms ofsensory receptors are envisioned in this disclosure. Furthermore, theprinciples disclosed herein for energy propagation along propagationpaths may be applicable to both energy emission and energy capture.

Many technologies exist today that are often unfortunately confused withholograms including lenticular printing, Pepper's Ghost, glasses-freestereoscopic displays, horizontal parallax displays, head-mounted VR andAR displays (HMD), and other such illusions generalized as“fauxlography.” These technologies may exhibit some of the desiredproperties of a true holographic display, however, lack the ability tostimulate the human visual sensory response in any way sufficient toaddress at least two of the four identified Holodeck Design Parameters.

These challenges have not been successfully implemented by conventionaltechnology to produce a seamless energy surface sufficient forholographic energy propagation. There are various approaches toimplementing volumetric and direction multiplexed light field displaysincluding parallax barriers, hogels, voxels, diffractive optics,multi-view projection, holographic diffusers, rotational mirrors,multilayered displays, time sequential displays, head mounted display,etc., however, conventional approaches may involve a compromise on imagequality, resolution, angular sampling density, size, cost, safety, framerate, etc., ultimately resulting in an unviable technology.

To achieve the Holodeck Design Parameters for the visual, auditory,somatosensory systems, the human acuity of each of the respectivesystems is studied and understood to propagate energy waves tosufficiently fool the human sensory receptors. The visual system iscapable of resolving to approximately 1 arc min, the auditory system maydistinguish the difference in placement as little as three degrees, andthe somatosensory system at the hands are capable of discerning pointsseparated by 2-12 mm. While there are various and conflicting ways tomeasure these acuities, these values are sufficient to understand thesystems and methods to stimulate perception of energy propagation.

Of the noted sensory receptors, the human visual system is by far themost sensitive given that even a single photon can induce sensation. Forthis reason, much of this introduction will focus on visual energy wavepropagation, and vastly lower resolution energy systems coupled within adisclosed energy waveguide surface may converge appropriate signals toinduce holographic sensory perception. Unless otherwise noted, alldisclosures apply to all energy and sensory domains.

When calculating for effective design parameters of the energypropagation for the visual system given a viewing volume and viewingdistance, a desired energy surface may be designed to include manygigapixels of effective energy location density. For wide viewingvolumes, or near field viewing, the design parameters of a desiredenergy surface may include hundreds of gigapixels or more of effectiveenergy location density. By comparison, a desired energy source may bedesigned to have 1 to 250 effective megapixels of energy locationdensity for ultrasonic propagation of volumetric haptics or an array of36 to 3,600 effective energy locations for acoustic propagation ofholographic sound depending on input environmental variables. What isimportant to note is that with a disclosed bi-directional energy surfacearchitecture, all components may be configured to form the appropriatestructures for any energy domain to enable holographic propagation.

However, the main challenge to enable the Holodeck today involvesavailable visual technologies and energy device limitations. Acousticand ultrasonic devices are less challenging given the orders ofmagnitude difference in desired density based upon sensory acuity in therespective receptive field, although the complexity should not beunderestimated. While holographic emulsion exists with resolutionsexceeding the desired density to encode interference patterns in staticimagery, state-of-the-art display devices are limited by resolution,data throughput and manufacturing feasibility. To date, no singulardisplay device has been able to meaningfully produce a light fieldhaving near holographic resolution for visual acuity.

Production of a single silicon-based device capable of meeting thedesired resolution for a compelling light field display may notpractical and may involve extremely complex fabrication processes beyondthe current manufacturing capabilities. The limitation to tilingmultiple existing display devices together involves the seams and gapformed by the physical size of packaging, electronics, enclosure, opticsand a number of other challenges that inevitably result in an unviabletechnology from an imaging, cost and/or a size standpoint.

The embodiments disclosed herein may provide a real-world path tobuilding the Holodeck.

Example embodiments will now be described hereinafter with reference tothe accompanying drawings, which form a part hereof, and whichillustrate example embodiments which may be practiced. As used in thedisclosures and the appended claims, the terms “embodiment”, “exampleembodiment”, and “exemplary embodiment” do not necessarily refer to asingle embodiment, although they may, and various example embodimentsmay be readily combined and interchanged, without departing from thescope or spirit of example embodiments. Furthermore, the terminology asused herein is for the purpose of describing example embodiments onlyand is not intended to be limitations. In this respect, as used herein,the term “in” may include “in” and “on”, and the terms “a,” “an” and“the” may include singular and plural references. Furthermore, as usedherein, the term “by” may also mean “from”, depending on the context.Furthermore, as used herein, the term “if” may also mean “when” or“upon,” depending on the context. Furthermore, as used herein, the words“and/or” may refer to and encompass any and all possible combinations ofone or more of the associated listed items.

Holographic System Considerations Overview of Light Field EnergyPropagation Resolution

Light field and holographic display is the result of a plurality ofprojections where energy surface locations provide angular, color andintensity information propagated within a viewing volume. The disclosedenergy surface provides opportunities for additional information tocoexist and propagate through the same surface to induce other sensorysystem responses. Unlike a stereoscopic display, the viewed position ofthe converged energy propagation paths in space do not vary as theviewer moves around the viewing volume and any number of viewers maysimultaneously see propagated objects in real-world space as if it wastruly there. In some embodiments, the propagation of energy may belocated in the same energy propagation path but in opposite directions.For example, energy emission and energy capture along an energypropagation path are both possible in some embodiments of the presentdisclosed.

FIG. 1 is a schematic diagram illustrating variables relevant forstimulation of sensory receptor response. These variables may includesurface diagonal 101, surface width 102, surface height 103, adetermined target seating distance 118, the target seating field of viewfield of view from the center of the display 104, the number ofintermediate samples demonstrated here as samples between the eyes 105,the average adult inter-ocular separation 106, the average resolution ofthe human eye in arcmin 107, the horizontal field of view formed betweenthe target viewer location and the surface width 108, the vertical fieldof view formed between the target viewer location and the surface height109, the resultant horizontal waveguide element resolution, or totalnumber of elements, across the surface 110, the resultant verticalwaveguide element resolution, or total number of elements, across thesurface 111, the sample distance based upon the inter-ocular spacingbetween the eyes and the number of intermediate samples for angularprojection between the eyes 112, the angular sampling may be based uponthe sample distance and the target seating distance 113, the totalresolution Horizontal per waveguide element derived from the angularsampling desired 114, the total resolution Vertical per waveguideelement derived from the angular sampling desired 115, device Horizontalis the count of the determined number of discreet energy sources desired116, and device Vertical is the count of the determined number ofdiscreet energy sources desired 117.

A method to understand the desired minimum resolution may be based uponthe following criteria to ensure sufficient stimulation of visual (orother) sensory receptor response: surface size (e.g., 84″ diagonal),surface aspect ratio (e.g., 16:9), seating distance (e.g., 128″ from thedisplay), seating field of view (e.g., 120 degrees or +/−60 degreesabout the center of the display), desired intermediate samples at adistance (e.g., one additional propagation path between the eyes), theaverage inter-ocular separation of an adult (approximately 65 mm), andthe average resolution of the human eye (approximately 1 arcmin). Theseexample values should be considered placeholders depending on thespecific application design parameters.

Further, each of the values attributed to the visual sensory receptorsmay be replaced with other systems to determine desired propagation pathparameters. For other energy propagation embodiments, one may considerthe auditory system's angular sensitivity as low as three degrees, andthe somatosensory system's spatial resolution of the hands as small as2-12 mm.

While there are various and conflicting ways to measure these sensoryacuities, these values are sufficient to understand the systems andmethods to stimulate perception of virtual energy propagation. There aremany ways to consider the design resolution, and the below proposedmethodology combines pragmatic product considerations with thebiological resolving limits of the sensory systems. As will beappreciated by one of ordinary skill in the art, the following overviewis a simplification of any such system design, and should be consideredfor exemplary purposes only.

With the resolution limit of the sensory system understood, the totalenergy waveguide element density may be calculated such that thereceiving sensory system cannot discern a single energy waveguideelement from an adjacent element, given:

${{Surface}\mspace{14mu}{Aspect}\mspace{14mu}{Ratio}} = \frac{{Width}\mspace{14mu}(W)}{{Height}\mspace{14mu}(H)}$${{Surface}\mspace{14mu}{Horizontal}\mspace{14mu}{Size}} = {{Surface}\mspace{14mu}{Diagonal}*\left( \frac{1}{\sqrt{\left( {1 + \left( \frac{H}{W} \right)^{2}} \right.}} \right)}$${{Surface}\mspace{14mu}{Vertical}\mspace{14mu}{Size}} = {{Surface}\mspace{14mu}{Diagonal}*\left( \frac{1}{\sqrt{\left( {1 + \left( \frac{W}{H} \right)^{2}} \right.}} \right)}$${{Horizontal}\mspace{14mu}{Field}\mspace{14mu}{of}\mspace{14mu}{View}} = {2*{{atan}\left( \frac{{Surface}\mspace{14mu}{Horizontal}\mspace{14mu}{Size}}{2*{Seating}\mspace{14mu}{Distance}} \right)}}$${{Vertical}\mspace{14mu}{Field}\mspace{14mu}{of}\mspace{14mu}{View}} = {2*{{atan}\left( \frac{{Surface}\mspace{14mu}{Vertical}\mspace{14mu}{Size}}{2*{Seating}\mspace{14mu}{Distance}} \right)}}$${{Horizontal}\mspace{14mu}{Element}\mspace{14mu}{Resolution}} = {{Horizontal}\mspace{14mu}{FoV}*\frac{60}{{Eye}\mspace{14mu}{Resolultion}}}$${{Vertical}\mspace{14mu}{Element}\mspace{14mu}{Resolution}} = {{Vertical}\mspace{14mu}{FoV}*\frac{60}{{Eye}\mspace{14mu}{Resolultion}}}$

The above calculations result in approximately a 32×18° field of viewresulting in approximately 1920×1080 (rounded to nearest format) energywaveguide elements being desired. One may also constrain the variablessuch that the field of view is consistent for both (u, v) to provide amore regular spatial sampling of energy locations (e.g. pixel aspectratio). The angular sampling of the system assumes a defined targetviewing volume location and additional propagated energy paths betweentwo points at the optimized distance, given:

${{Sample}\mspace{14mu}{Distance}} = \frac{{Inter}\text{-}{Ocular}\mspace{14mu}{Distance}}{\left( {{{Number}\mspace{14mu}{of}\mspace{14mu}{Desired}\mspace{14mu}{Intermediate}\mspace{14mu}{Samples}} + 1} \right)}$${{Angular}\mspace{14mu}{Sampling}} = {{atan}\left( \frac{{Sample}\mspace{14mu}{Distance}}{{Seating}\mspace{14mu}{Distance}} \right)}$

In this case, the inter-ocular distance is leveraged to calculate thesample distance although any metric may be leveraged to account forappropriate number of samples as a given distance. With the abovevariables considered, approximately one ray per 0.57° may be desired andthe total system resolution per independent sensory system may bedetermined, given:

${{Locations}\mspace{14mu}{Per}\mspace{14mu}{Element}\mspace{14mu}(N)} = \frac{{Seating}\mspace{14mu}{FoV}}{{Angular}\mspace{14mu}{Sampling}}$Total  Resolution  H = N * Horizontal  Element  ResolutionTotal  Resolution  V = N * Vertical  Element  Resolution

With the above scenario given the size of energy surface and the angularresolution addressed for the visual acuity system, the resultant energysurface may desirably include approximately 400 k×225 k pixels of energyresolution locations, or 90 gigapixels holographic propagation density.These variables provided are for exemplary purposes only and many othersensory and energy metrology considerations should be considered for theoptimization of holographic propagation of energy. In an additionalembodiment, 1 gigapixel of energy resolution locations may be desiredbased upon the input variables. In an additional embodiment, 1,000gigapixels of energy resolution locations may be desired based upon theinput variables.

Current Technology Limitations Active Area, Device Electronics,Packaging, and the Mechanical Envelope

FIG. 2 illustrates a device 200 having an active area 220 with a certainmechanical form factor. The device 200 may include drivers 230 andelectronics 240 for powering and interface to the active area 220, theactive area having a dimension as shown by the x and y arrows. Thisdevice 200 does not take into account the cabling and mechanicalstructures to drive, power and cool components, and the mechanicalfootprint may be further minimized by introducing a flex cable into thedevice 200. The minimum footprint for such a device 200 may also bereferred to as a mechanical envelope 210 having a dimension as shown bythe M:x and M:y arrows. This device 200 is for illustration purposesonly and custom electronics designs may further decrease the mechanicalenvelope overhead, but in almost all cases may not be the exact size ofthe active area of the device. In an embodiment, this device 200illustrates the dependency of electronics as it relates to active imagearea 220 for a micro OLED, DLP chip or LCD panel, or any othertechnology with the purpose of image illumination.

In some embodiments, it may also be possible to consider otherprojection technologies to aggregate multiple images onto a largeroverall display. However, this may come at the cost of greatercomplexity for throw distance, minimum focus, optical quality, uniformfield resolution, chromatic aberration, thermal properties, calibration,alignment, additional size or form factor. For most practicalapplications, hosting tens or hundreds of these projection sources 200may result in a design that is much larger with less reliability.

For exemplary purposes only, assuming energy devices with an energylocation density of 3840×2160 sites, one may determine the number ofindividual energy devices (e.g., device 100) desired for an energysurface, given:

${{Devices}\mspace{14mu} H} = \frac{{Total}\mspace{14mu}{Resolution}\mspace{14mu} H}{{Device}\mspace{14mu}{Resolution}\mspace{14mu} H}$${{Devices}\mspace{14mu} V} = \frac{{Total}\mspace{14mu}{Resolution}\mspace{14mu} V}{{Device}\mspace{14mu}{Resolution}\mspace{14mu} V}$

Given the above resolution considerations, approximately 105×105 devicessimilar to those shown in FIG. 2 may be desired. It should be noted thatmany devices consist of various pixel structures that may or may not mapto a regular grid. In the event that there are additional sub-pixels orlocations within each full pixel, these may be exploited to generateadditional resolution or angular density. Additional signal processingmay be used to determine how to convert the light field into the correct(u,v) coordinates depending on the specified location of the pixelstructure(s) and can be an explicit characteristic of each device thatis known and calibrated. Further, other energy domains may involve adifferent handling of these ratios and device structures, and thoseskilled in the art will understand the direct intrinsic relationshipbetween each of the desired frequency domains. This will be shown anddiscussed in more detail in subsequent disclosure.

The resulting calculation may be used to understand how many of theseindividual devices may be desired to produce a full resolution energysurface. In this case, approximately 105×105 or approximately 11,080devices may be desired to achieve the visual acuity threshold. Thechallenge and novelty exists within the fabrication of a seamless energysurface from these available energy locations for sufficient sensoryholographic propagation.

Summary of Seamless Energy Surfaces Configurations and Designs forArrays of Energy Relays

In some embodiments, approaches are disclosed to address the challengeof generating high energy location density from an array of individualdevices without seams due to the limitation of mechanical structure forthe devices. In an embodiment, an energy propagating relay system mayallow for an increase the effective size of the active device area tomeet or exceed the mechanical dimensions to configure an array of relaysand form a singular seamless energy surface.

FIG. 3 illustrates an embodiment of such an energy relay system 300. Asshown, the relay system 300 may include a device 310 mounted to amechanical envelope 320, with an energy relay element 330 propagatingenergy from the device 310. The relay element 330 may be configured toprovide the ability to mitigate any gaps 340 that may be produced whenmultiple mechanical envelopes 320 of the device are placed into an arrayof multiple devices 310.

For example, if a device's active area 310 is 20 mm×10 mm and themechanical envelope 320 is 40 mm×20 mm, an energy relay element 330 maybe designed with a magnification of 2:1 to produce a tapered form thatis approximately 20 mm×10 mm on a minified end (arrow A) and 40 mm×20 mmon a magnified end (arrow B), providing the ability to align an array ofthese elements 330 together seamlessly without altering or collidingwith the mechanical envelope 320 of each device 310. Mechanically, therelay elements 330 may be bonded or fused together to align and polishensuring minimal seam gap 340 between devices 310. In one suchembodiment, it is possible to achieve a seam gap 340 smaller than thevisual acuity limit of the eye.

FIG. 4 illustrates an example of a base structure 400 having energyrelay elements 410 formed together and securely fastened to anadditional mechanical structure 430. The mechanical structure of theseamless energy surface 420 provides the ability to couple multipleenergy relay elements 410, 450 in series to the same base structurethrough bonding or other mechanical processes to mount relay elements410, 450. In some embodiments, each relay element 410 may be fused,bonded, adhered, pressure fit, aligned or otherwise attached together toform the resultant seamless energy surface 420. In some embodiments, adevice 480 may be mounted to the rear of the relay element 410 andaligned passively or actively to ensure appropriate energy locationalignment within the determined tolerance is maintained.

In an embodiment, the seamless energy surface comprises one or moreenergy locations and one or more energy relay element stacks comprise afirst and second side and each energy relay element stack is arranged toform a singular seamless energy surface directing energy alongpropagation paths extending between one or more energy locations and theseamless energy surface, and where the separation between the edges ofany two adjacent second sides of the terminal energy relay elements isless than the minimum perceptible contour as defined by the visualacuity of a human eye having better than 20/40 vision at a distancegreater than the width of the singular seamless energy surface.

In an embodiment, each of the seamless energy surfaces comprise one ormore energy relay elements each with one or more structures forming afirst and second surface with a transverse and longitudinal orientation.The first relay surface has an area different than the second resultingin positive or negative magnification and configured with explicitsurface contours for both the first and second surfaces passing energythrough the second relay surface to substantially fill a +/−10 degreeangle with respect to the normal of the surface contour across theentire second relay surface.

In an embodiment, multiple energy domains may be configured within asingle, or between multiple energy relays to direct one or more sensoryholographic energy propagation paths including visual, acoustic, tactileor other energy domains.

In an embodiment, the seamless energy surface is configured with energyrelays that comprise two or more first sides for each second side toboth receive and emit one or more energy domains simultaneously toprovide bi-directional energy propagation throughout the system.

In an embodiment, the energy relays are provided as loose coherentelements.

Introduction to Component Engineered Structures Disclosed Advances inTransverse Anderson Localization Energy Relays

The properties of energy relays may be significantly optimized accordingto the principles disclosed herein for energy relay elements that induceTransverse Anderson Localization. Transverse Anderson Localization isthe propagation of a ray transported through a transversely disorderedbut longitudinally consistent material.

This implies that the effect of the materials that produce the AndersonLocalization phenomena may be less impacted by total internal reflectionthan by the randomization between multiple-scattering paths where waveinterference can completely limit the propagation in the transverseorientation while continuing in the longitudinal orientation.

Of significant additional benefit is the elimination of the cladding oftraditional multi-core optical fiber materials. The cladding is tofunctionally eliminate the scatter of energy between fibers, butsimultaneously act as a barrier to rays of energy thereby reducingtransmission by at least the core to clad ratio (e.g., a core to cladratio of 70:30 will transmit at best 70% of received energytransmission) and additionally forms a strong pixelated patterning inthe propagated energy.

FIG. 5A illustrates an end view of an example of one such non-AndersonLocalization energy relay 500, wherein an image is relayed throughmulti-core optical fibers where pixilation and fiber noise may beexhibited due to the intrinsic properties of the optical fibers. Withtraditional multi-mode and multi-core optical fibers, relayed images maybe intrinsically pixelated due to the properties of total internalreflection of the discrete array of cores where any cross-talk betweencores will reduce the modulation transfer function and increaseblurring. The resulting imagery produced with traditional multi-coreoptical fiber tends to have a residual fixed noise fiber pattern similarto those shown in FIG. 3 .

FIG. 5B, illustrates an example of the same relayed image 550 through anenergy relay comprising materials that exhibit the properties ofTransverse Anderson Localization, where the relayed pattern has agreater density grain structures as compared to the fixed fiber patternfrom FIG. 5A. In an embodiment, relays comprising randomized microscopiccomponent engineered structures induce Transverse Anderson Localizationand transport light more efficiently with higher propagation ofresolvable resolution than commercially available multi-mode glassoptical fibers.

There is significant advantage to the Transverse Anderson Localizationmaterial properties in terms of both cost and weight, where a similaroptical grade glass material, may cost and weigh upwards of 10 to100-fold more than the cost for the same material generated within anembodiment, wherein disclosed systems and methods comprise randomizedmicroscopic component engineered structures demonstrating significantopportunities to improve both cost and quality over other technologiesknown in the art.

In an embodiment, a relay element exhibiting Transverse AndersonLocalization may comprise a plurality of at least two differentcomponent engineered structures in each of three orthogonal planesarranged in a dimensional lattice and the plurality of structures formrandomized distributions of material wave propagation properties in atransverse plane within the dimensional lattice and channels of similarvalues of material wave propagation properties in a longitudinal planewithin the dimensional lattice, wherein localized energy wavespropagating through the energy relay have higher transport efficiency inthe longitudinal orientation versus the transverse orientation.

In an embodiment, multiple energy domains may be configured within asingle, or between multiple Transverse Anderson Localization energyrelays to direct one or more sensory holographic energy propagationpaths including visual, acoustic, tactile or other energy domains.

In an embodiment, the seamless energy surface is configured withTransverse Anderson Localization energy relays that comprise two or morefirst sides for each second side to both receive and emit one or moreenergy domains simultaneously to provide bi-directional energypropagation throughout the system.

In an embodiment, the Transverse Anderson Localization energy relays areconfigured as loose coherent or flexible energy relay elements.

Considerations for 4D Plenoptic Functions Selective Propagation ofEnergy Through Holographic Waveguide Arrays

As discussed above and herein throughout, a light field display systemgenerally includes an energy source (e.g., illumination source) and aseamless energy surface configured with sufficient energy locationdensity as articulated in the above discussion. A plurality of relayelements may be used to relay energy from the energy devices to theseamless energy surface. Once energy has been delivered to the seamlessenergy surface with the requisite energy location density, the energycan be propagated in accordance with a 4D plenoptic function through adisclosed energy waveguide system. As will be appreciated by one ofordinary skill in the art, a 4D plenoptic function is well known in theart and will not be elaborated further herein.

The energy waveguide system selectively propagates energy through aplurality of energy locations along the seamless energy surfacerepresenting the spatial coordinate of the 4D plenoptic function with astructure configured to alter an angular direction of the energy wavespassing through representing the angular component of the 4D plenopticfunction, wherein the energy waves propagated may converge in space inaccordance with a plurality of propagation paths directed by the 4Dplenoptic function.

Reference is now made to FIG. 6 illustrating an example of light fieldenergy surface in 4D image space in accordance with a 4D plenopticfunction. The figure shows ray traces of an energy surface 600 to aviewer 620 in describing how the rays of energy converge in space 630from various positions within the viewing volume. As shown, eachwaveguide element 610 defines four dimensions of information describingenergy propagation 640 through the energy surface 600. Two spatialdimensions (herein referred to as x and y) are the physical plurality ofenergy locations that can be viewed in image space, and the angularcomponents theta and phi (herein referred to as u and v), which isviewed in virtual space when projected through the energy waveguidearray. In general, and in accordance with a 4D plenoptic function, theplurality of waveguides (e.g., lenslets) are able to direct an energylocation from the x, y dimension to a unique location in virtual space,along a direction defined by the u, v angular component, in forming theholographic or light field system described herein.

However, one skilled in the art will understand that a significantchallenge to light field and holographic display technologies arisesfrom uncontrolled propagation of energy due designs that have notaccurately accounted for any of diffraction, scatter, diffusion, angulardirection, calibration, focus, collimation, curvature, uniformity,element cross-talk, as well as a multitude of other parameters thatcontribute to decreased effective resolution as well as an inability toaccurately converge energy with sufficient fidelity.

In an embodiment, an approach to selective energy propagation foraddressing challenges associated with holographic display may includeenergy inhibiting elements and substantially filling waveguide apertureswith near-collimated energy into an environment defined by a 4Dplenoptic function.

In an embodiment, an array of energy waveguides may define a pluralityof energy propagation paths for each waveguide element configured toextend through and substantially fill the waveguide element's effectiveaperture in unique directions defined by a prescribed 4D function to aplurality of energy locations along a seamless energy surface inhibitedby one or more elements positioned to limit propagation of each energylocation to only pass through a single waveguide element.

In an embodiment, multiple energy domains may be configured within asingle, or between multiple energy waveguides to direct one or moresensory holographic energy propagations including visual, acoustic,tactile or other energy domains.

In an embodiment, the energy waveguides and seamless energy surface areconfigured to both receive and emit one or more energy domains toprovide bi-directional energy propagation throughout the system.

In an embodiment, the energy waveguides are configured to propagatenon-linear or non-regular distributions of energy, includingnon-transmitting void regions, leveraging digitally encoded,diffractive, refractive, reflective, grin, holographic, Fresnel, or thelike waveguide configurations for any seamless energy surfaceorientation including wall, table, floor, ceiling, room, or othergeometry based environments. In an additional embodiment, an energywaveguide element may be configured to produce various geometries thatprovide any surface profile and/or tabletop viewing allowing users toview holographic imagery from all around the energy surface in a360-degree configuration.

In an embodiment, the energy waveguide array elements may be reflectivesurfaces and the arrangement of the elements may be hexagonal, square,irregular, semi-regular, curved, non-planar, spherical, cylindrical,tilted regular, tilted irregular, spatially varying and/ormulti-layered.

For any component within the seamless energy surface, waveguide, orrelay components may include, but not limited to, optical fiber,silicon, glass, polymer, optical relays, diffractive, holographic,refractive, or reflective elements, optical face plates, energycombiners, beam splitters, prisms, polarization elements, spatial lightmodulators, active pixels, liquid crystal cells, transparent displays,or any similar materials exhibiting Anderson localization or totalinternal reflection.

Realizing the Holodeck Aggregation of Bi-Directional Seamless EnergySurface Systems to Stimulate Human Sensory Receptors within HolographicEnvironments

It is possible to construct large-scale environments of seamless energysurface systems by tiling, fusing, bonding, attaching, and/or stitchingmultiple seamless energy surfaces together forming arbitrary sizes,shapes, contours or form-factors including entire rooms. Each energysurface system may comprise an assembly having a base structure, energysurface, relays, waveguide, devices, and electronics, collectivelyconfigured for bi-directional holographic energy propagation, emission,reflection, or sensing.

In an embodiment, an environment of tiled seamless energy systems isaggregated to form large seamless planar or curved walls includinginstallations comprising up to all surfaces in a given environment, andconfigured as any combination of seamless, discontinuous planar,faceted, curved, cylindrical, spherical, geometric, or non-regulargeometries.

In an embodiment, aggregated tiles of planar surfaces form wall-sizedsystems for theatrical or venue-based holographic entertainment. In anembodiment, aggregated tiles of planar surfaces cover a room with fourto six walls including both ceiling and floor for cave-based holographicinstallations. In an embodiment, aggregated tiles of curved surfacesproduce a cylindrical seamless environment for immersive holographicinstallations. In an embodiment, aggregated tiles of seamless sphericalsurfaces form a holographic dome for immersive Holodeck-basedexperiences.

In an embodiment, aggregates tiles of seamless curved energy waveguidesprovide mechanical edges following the precise pattern along theboundary of energy inhibiting elements within the energy waveguidestructure to bond, align, or fuse the adjacent tiled mechanical edges ofthe adjacent waveguide surfaces, resulting in a modular and seamlessenergy waveguide system.

In a further embodiment of an aggregated tiled environment, energy ispropagated bi-directionally for multiple simultaneous energy domains. Inan additional embodiment, the energy surface provides the ability toboth display and capture simultaneously from the same energy surfacewith waveguides designed such that light field data may be projected byan illumination source through the waveguide and simultaneously receivedthrough the same energy surface. In an additional embodiment, additionaldepth sensing and active scanning technologies may be leveraged to allowfor the interaction between the energy propagation and the viewer incorrect world coordinates. In an additional embodiment, the energysurface and waveguide are operable to emit, reflect or convergefrequencies to induce tactile sensation or volumetric haptic feedback.In some embodiments, any combination of bi-directional energypropagation and aggregated surfaces are possible.

In an embodiment, the system comprises an energy waveguide capable ofbi-directional emission and sensing of energy through the energy surfacewith one or more energy devices independently paired withtwo-or-more-path energy combiners to pair at least two energy devices tothe same portion of the seamless energy surface, or one or more energydevices are secured behind the energy surface, proximate to anadditional component secured to the base structure, or to a location infront and outside of the FOV of the waveguide for off-axis direct orreflective projection or sensing, and the resulting energy surfaceprovides for bi-directional transmission of energy allowing thewaveguide to converge energy, a first device to emit energy and a seconddevice to sense energy, and where the information is processed toperform computer vision related tasks including, but not limited to, 4Dplenoptic eye and retinal tracking or sensing of interference withinpropagated energy patterns, depth estimation, proximity, motiontracking, image, color, or sound formation, or other energy frequencyanalysis. In an additional embodiment, the tracked positions activelycalculate and modify positions of energy based upon the interferencebetween the bi-directional captured data and projection information.

In some embodiments, a plurality of combinations of three energy devicescomprising an ultrasonic sensor, a visible energy display, and anultrasonic emitting device are configured together for each of threefirst relay surfaces propagating energy combined into a single secondenergy relay surface with each of the three first surfaces comprisingengineered properties specific to each device's energy domain, and twoengineered waveguide elements configured for ultrasonic and energyrespectively to provide the ability to direct and converge each device'senergy independently and substantially unaffected by the other waveguideelements that are configured for a separate energy domain.

In some embodiments, disclosed is a calibration procedure to enableefficient manufacturing to remove system artifacts and produce ageometric mapping of the resultant energy surface for use withencoding/decoding technologies as well as dedicated integrated systemsfor the conversion of data into calibrated information appropriate forenergy propagation based upon the calibrated configuration files.

In some embodiments, additional energy waveguides in series and one ormore energy devices may be integrated into a system to produce opaqueholographic pixels.

In some embodiments, additional waveguide elements may be integratedcomprising energy inhibiting elements, beam-splitters, prisms, activeparallax barriers or polarization technologies in order to providespatial and/or angular resolutions greater than the diameter of thewaveguide or for other super-resolution purposes.

In some embodiments, the disclosed energy system may also be configuredas a wearable bi-directional device, such as virtual reality (VR) oraugmented reality (AR). In other embodiments, the energy system mayinclude adjustment optical element(s) that cause the displayed orreceived energy to be focused proximate to a determined plane in spacefor a viewer. In some embodiments, the waveguide array may beincorporated to holographic head-mounted-display. In other embodiments,the system may include multiple optical paths to allow for the viewer tosee both the energy system and a real-world environment (e.g.,transparent holographic display). In these instances, the system may bepresented as near field in addition to other methods.

In some embodiments, the transmission of data comprises encodingprocesses with selectable or variable compression ratios that receive anarbitrary dataset of information and metadata; analyze said dataset andreceive or assign material properties, vectors, surface IDs, new pixeldata forming a more sparse dataset, and wherein the received data maycomprise: 2D, stereoscopic, multi-view, metadata, light field,holographic, geometry, vectors or vectorized metadata, and anencoder/decoder may provide the ability to convert the data in real-timeor off-line comprising image processing for: 2D; 2D plus depth, metadataor other vectorized information; stereoscopic, stereoscopic plus depth,metadata or other vectorized information; multi-view; multi-view plusdepth, metadata or other vectorized information; holographic; or lightfield content; through depth estimation algorithms, with or withoutdepth metadata; and an inverse ray tracing methodology appropriatelymaps the resulting converted data produced by inverse ray tracing fromthe various 2D, stereoscopic, multi-view, volumetric, light field orholographic data into real world coordinates through a characterized 4Dplenoptic function. In these embodiments, the total data transmissiondesired may be multiple orders of magnitudes less transmittedinformation than the raw light field dataset.

Energy Directing Devices Suitable for Presenting Holographic SensoryData

In an embodiment, the optomechanical display device may be capable ofemitting and guiding light to form 2D, stereoscopic, multiview,plenoptic, 4D, volumetric, light field, holographic, or any other visualrepresentation of light.

FIG. 7A is an example of a light field optomechanical system ifconfigured with emissive display devices, optical relays, and awaveguide that is realized as an array of refractive elements such as amicro lens array, where a visible image from one or more displays may beoptically relayed before being transmitted to the energy surface, wherethe array of refractive elements provides a mapping between eachlocation on the energy surface and the direction of projection of thelight from that location, such that a 4D volumetric light field imagemay be projected.

In an embodiment, the waveguide may be operable to converge rays oflight to induce both vergence and accommodation from an observer pointof view.

In an embodiment, the waveguides and energy relays may be formed orpolished with various surface geometries. In an embodiment, the energyrelays include elements that induce transverse Anderson localization. Inan embodiment, the energy relays are bidirectional and may both emitand/or project energy.

In one embodiment, an energy system configured to direct energyaccording to a four-dimensional (4D) plenoptic function includes aplurality of energy devices. In some embodiments, the plurality ofenergy devices include illumination sources emitting image information,where the image information includes emissive, projection, or reflectivedisplay technologies, leveraging visible, IR, UV, coherent, laser,infrared, polarized or any other electromagnetic illumination source. Inother embodiments, the plurality of energy devices include mechanicalenergy emitting devices configured to provide immersive audio orvolumetric tactile sensation from an acoustic field.

In some embodiments, the energy system as configured above may furtherinclude a base structure (e.g., 72) such that the plurality of energydevices, the energy relay system, and the energy waveguide system mayall be coupled to the base structure. In other embodiments, theplurality of energy devices, the energy relay system and the energywaveguide system may be coupled to the base structure with one or moremounting brackets.

In some embodiments, the plurality of energy devices include energydevices for capturing or sensing energy, including mechanical, chemical,transfer, thermal, electric, potential, kinetic, magnetic,gravitational, radiant, energy, structured, unstructured, or other formsof energy. In other embodiments, the plurality of energy devices includeenergy devices for propagating or emitting energy, including mechanical,chemical, transfer, thermal, electric, potential, kinetic, magnetic,gravitational, radiant, energy, structured, unstructured, or other formsof energy. In yet other embodiments, the plurality of energy devicesinclude acoustic receiving devices configured to provide sensoryfeedback or audible controls

In one embodiment, the energy system further includes an energy relaysystem (e.g., 6110 as best shown in FIG. 7B) having one or more energyrelay elements, where each of the one or more energy relay elementsincludes a first surface and a second surface, the second surface of theone or more energy relay elements being arranged to form a singularseamless energy surface of the energy relay system, and where a firstplurality of energy propagation paths extend from the energy locationsin the plurality of energy devices through the singular seamless energysurface of the energy relay system. This will be discussed in moredetail below.

Reference is now made to FIG. 7B illustrating an energy relay system6110, in an orthogonal view in accordance with one embodiment of thepresent disclosure. In one embodiment, the energy relay system 6110 mayinclude two or more relay elements 6112, each relay element 6112 formedof one or more structures, each relay element 6112 having a firstsurface 6114, a second surface 6116, a transverse orientation (generallyparallel to the surfaces 6114, 6116) and a longitudinal orientation(generally perpendicular to the surfaces 6114, 6116). In one embodiment,the surface area of the first surface 6114 may be different than thesurface area of the second surface 6116. For example, the surface areaof the first surface 6114 may be greater or lesser than the surface areaof the second surface 6116. In another embodiment, the surface area ofthe first surface 114 may be the same as the surface area of the secondsurface 6116. Energy waves can pass from the first surface 6114 to thesecond surface 6116, or vice versa.

In one embodiment, the relay element 6112 of the energy relay system6110 includes a sloped profile portion 6118 between the first surface6114 and the second surface 6116. In operation, energy waves propagatingbetween the first surface 6114 and the second surface 6116 may have ahigher transport efficiency in the longitudinal orientation than in thetransverse orientation, and energy waves passing through the relayelement 6112 may result in spatial magnification or spatialde-magnification. In other words, energy waves passing through the relayelement 6112 of the relay element device 6110 may experience increasedmagnification or decreased magnification. In some embodiments, the oneor more structures for forming the energy relay element 6110 may includeglass, carbon, optical fiber, optical film, plastic, polymer, ormixtures thereof.

In one embodiment, the energy waves passing through the first surface6114 has a first resolution, while the energy waves passing through thesecond surface 6116 has a second resolution, and the second resolutionis no less than about 50% of the first resolution. In anotherembodiment, the energy waves, while having a uniform profile whenpresented to the first surface, may pass through the second surfaceradiating in every direction with an energy density in the forwarddirection that substantially fills a cone with an opening angle of +/−10degrees relative to the normal to the second surface, irrespective oflocation on the second relay surface.

In some embodiments, the first surface 6114 may be configured to receiveenergy from an energy wave source, the energy wave source including amechanical envelope having a width different than the width of at leastone of the first surface 6114 and the second surface 6116.

In each relay 6112, energy is transported between first and secondsurfaces which defines the longitudinal orientation, the first andsecond surfaces of each of the relays extends generally along atransverse orientation defined by the first and second directions, wherethe longitudinal orientation is substantially normal to the transverseorientation. In one embodiment, energy waves propagating through theplurality of relays have higher transport efficiency in the longitudinalorientation than in the transverse orientation due to randomizedrefractive index variability in the transverse orientation coupled withminimal refractive index variation in the longitudinal orientation. Insome embodiments where each relay is constructed of multicore fiber, theenergy waves propagating within each relay element may travel in thelongitudinal orientation determined by the alignment of fibers in thisorientation.

In an embodiment, a separation between the edges of any two adjacentsecond sides of the terminal energy relay elements may be less than aminimum perceptible contour as defined by the visual acuity of a humaneye having better than 20/40 vision at a distance from the seamlessenergy surface that is greater than the lesser of a height of thesingular seamless energy surface or a width of the singular seamlessenergy surface.

In one embodiment, the plurality of energy relay elements in the stackedconfiguration may include a plurality of faceplates. In someembodiments, the plurality of faceplates may have different lengths orare loose coherent optical relays. In other embodiments, the pluralityof elements may have sloped profile portions similar to that of FIG. 7B,where the sloped profile portions may be angled, linear, curved,tapered, faceted or aligned at a non-perpendicular angle relative to anormal axis of the relay element. In yet another embodiment, energywaves propagating through the plurality of relay elements have highertransport efficiency in the longitudinal orientation than in thetransverse orientation due to randomized refractive index variability inthe transverse orientation coupled with minimal refractive indexvariation in the longitudinal orientation. In embodiments where eachenergy relay is constructed of multicore fiber, the energy wavespropagating within each relay element may travel in the longitudinalorientation determined by the alignment of fibers in this orientation.

In some embodiments, the one or more relay elements (e.g., 6112)includes fused or tiled mosaics, where any seams between adjacent fusedor tiled mosaics are separated by or are less than the minimumperceptible contour as defined by the visual acuity of a human eyehaving better than 20/40 vision at a distance at or greater than thewidth or height of the singular seamless energy surface.

In other embodiments, the one or more relay elements (e.g., 6112)includes: optical fiber, silicon, glass, polymer, optical relays,diffractive elements, holographic optical elements, refractive elements,reflective elements, optical face plates, optical combiners, beamsplitters, prisms, polarization components, spatial light modulators,active pixels, liquid crystal cells, transparent displays, or anysimilar materials having Anderson localization or total internalreflection properties for forming the singular seamless energy surface.

In yet other embodiments, the one or more relay elements (e.g., 6112)are configured to accommodate a shape of the singular seamless energysurface including planar, spherical, cylindrical, conical, faceted,tiled, regular, non-regular, or any other geometric shape for aspecified application.

In another embodiment, the system further includes an energy waveguidesystem (e.g., 7100 as best shown in FIGS. 7C-7L) having an array ofenergy waveguides, where a second plurality of energy propagation pathsextend from the singular seamless energy surface through the array ofenergy waveguides in directions determined by a 4D plenoptic function.

FIG. 7C illustrates a top-down perspective view of an embodiment of anenergy waveguide system 7100 operable to define a plurality of energypropagation paths 7108. Energy waveguide system 7100 comprises an arrayof energy waveguides 7112 configured to direct energy therethrough alongthe plurality of energy propagation paths 7108. In an embodiment, theplurality of energy propagation paths 7108 extend through a plurality ofenergy locations 7118 on a first side of the array 7116 to a second sideof the array 7114.

Referring to FIGS. 7C and 7L, in an embodiment, a first subset 7290 ofthe plurality of energy propagation paths 7108 extend through a firstenergy location 7122. The first energy waveguide 7104 is configured todirect energy along a first energy propagation path 7120 of the firstsubset 7290 of the plurality of energy propagation paths 7108. The firstenergy propagation path 7120 may be defined by a first chief ray 7138formed between the first energy location 7122 and the first energywaveguide 7104. The first energy propagation path 7120 may comprise rays7138A and 7138B, formed between the first energy location 7122 and thefirst energy waveguide 7104, which are directed by first energywaveguide 7104 along energy propagation paths 7120A and 7120B,respectively. The first energy propagation path 7120 may extend from thefirst energy waveguide 7104 towards the second side of the array 7114.In an embodiment, energy directed along the first energy propagationpath 7120 comprises one or more energy propagation paths between orincluding energy propagation paths 7120A and 7120B, which are directedthrough the first energy waveguide 7104 in a direction that issubstantially parallel to the angle propagated through the second side7114 by the first chief ray 7138.

Embodiments may be configured such that energy directed along the firstenergy propagation path 7120 may exit the first energy waveguide 7104 ina direction that is substantially parallel to energy propagation paths7120A and 7120B and to the first chief ray 7138. It may be assumed thatan energy propagation path extending through an energy waveguide element7112 on the second side 7114 comprises a plurality of energy propagationpaths of a substantially similar propagation direction.

FIG. 7D is a front view illustration of an embodiment of energywaveguide system 7100. The first energy propagation path 7120 may extendtowards the second side of the array 7114 in a unique direction 7208extending from the first energy waveguide 7104, which is determined atleast by the first energy location 7122. The first energy waveguide 7104may be defined by a spatial coordinate 7204, and the unique direction7208 which is determined at least by first energy location 7122 may bedefined by an angular coordinate 7206 defining the directions of thefirst energy propagation path 7120. The spatial coordinate 7204 and theangular coordinate 7206 may form a four-dimensional plenoptic coordinateset 7210 which defines the unique direction 7208 of the first energypropagation path 7120.

In an embodiment, energy directed along the first energy propagationpath 7120 through the first energy waveguide 7104 substantially fills afirst aperture 7134 of the first energy waveguide 7104, and propagatesalong one or more energy propagation paths which lie between energypropagation paths 7120A and 7120B and are parallel to the direction ofthe first energy propagation path 7120. In an embodiment, the one ormore energy propagation paths that substantially fill the first aperture7134 may comprise greater than 50% of the first aperture 7134 diameter.

In a preferred embodiment, energy directed along the first energypropagation path 7120 through the first energy waveguide 7104 whichsubstantially fills the first aperture 7134 may comprise between 50% to80% of the first aperture 7134 diameter.

Turning back to FIGS. 7C and 7E-7L, in an embodiment, the energywaveguide system 7100 may further comprise an energy inhibiting element7124 positioned to limit propagation of energy between the first side7116 and the second side 7114 and to inhibit energy propagation betweenadjacent waveguides 7112. In an embodiment, the energy inhibitingelement is configured to inhibit energy propagation along a portion ofthe first subset 7290 of the plurality of energy propagation paths 7108that do not extend through the first aperture 7134. In an embodiment,the energy inhibiting element 7124 may be located on the first side 7116between the array of energy waveguides 7112 and the plurality of energylocations 7118. In an embodiment, the energy inhibiting element 7124 maybe located on the second side 7114 between the plurality of energylocations 7118 and the energy propagation paths 7108. In an embodiment,the energy inhibiting element 7124 may be located on the first side 7116or the second side 7114 orthogonal to the array of energy waveguides7112 or the plurality of energy locations 7118.

In an embodiment, energy directed along the first energy propagationpath 7120 may converge with energy directed along a second energypropagation path 7126 through a second energy waveguide 7128. The firstand second energy propagation paths may converge at a location 7130 onthe second side 7114 of the array 7112. In an embodiment, a third andfourth energy propagation paths 7140, 7141 may also converge at alocation 7132 on the first side 7116 of the array 7112. In anembodiment, a fifth and sixth energy propagation paths 7142, 7143 mayalso converge at a location 7136 between the first and second sides7116, 7114 of the array 7112.

FIGS. 7E-7L are illustrations of various embodiments of energyinhibiting element 7124. For the avoidance of doubt, these embodimentsare provided for exemplary purposes and in no way limiting to the scopeof the combinations or implementations provided within the scope of thisdisclosure.

FIG. 7E illustrates an embodiment of the plurality of energy locations7118 wherein an energy inhibiting element 7251 is placed adjacent to thesurface of the energy locations 7118 and comprises a specifiedrefractive, diffractive, reflective, or other energy altering property.The energy inhibiting element 7251 may be configured to limit the firstsubset of energy propagation paths 7290 to a smaller range ofpropagation paths 7253 by inhibiting propagation of energy along energypropagation paths 7252. In an embodiment, the energy inhibiting elementis an energy relay with a numerical aperture less than 1.

FIG. 7F illustrates an embodiment of the plurality of energy locations7118 wherein an energy inhibiting structure 7254 is placed orthogonalbetween regions of energy locations 7118, and wherein the energyinhibiting structure 7254 exhibits an absorptive property, and whereinthe inhibiting energy structure 7254 has a defined height along anenergy propagation path 7256 such that certain energy propagation paths7255 are inhibited. In an embodiment, the energy inhibiting structure7254 is hexagonal in shape. In an embodiment, the energy inhibitingstructure 7254 is round in shape. In an embodiment, the energyinhibiting structure 7254 is non-uniform in shape or size along anyorientation of the propagation path. In an embodiment, the energyinhibiting structure 7254 is embedded within another structure withadditional properties.

FIG. 7G illustrates the plurality of energy locations 7118, wherein afirst energy inhibiting structure 7257 is configured to substantiallyorient energy 7259 propagating therethrough into a first state. A secondenergy inhibiting structure 7258 is configured to allow energy 7259,which is substantially oriented in the first state, to propagatetherethrough, and to limit propagation of energy 7260 orientedsubstantially dissimilarly to the first state. In an embodiment, theenergy inhibiting element 7257, 7258 is an energy polarizing elementpair. In an embodiment, the energy inhibiting element 7257,7258 is anenergy wave band pass element pair. In an embodiment, the energyinhibiting element 7257, 7258 is a diffractive waveguide pair.

FIG. 7H illustrates an embodiment of the plurality of energy locations7118, wherein an energy inhibiting element 7261 is structured to alterenergy propagation paths 7263 to a certain extent depending upon whichof the plurality of energy locations 7118 the energy propagation paths7263 extends through. Energy inhibiting element 7261 may alter energypropagation paths 7263 in a uniform or non-uniform way along energypropagation paths 7263 such that certain energy propagation paths 7262are inhibited. An energy inhibiting structure 7254 is placed orthogonalbetween regions of energy locations 7118, and wherein the energyinhibiting structure 7254 exhibits an absorptive property, and whereinthe inhibiting energy structure 7254 has a defined height along anenergy propagation path 7263 such that certain energy propagation paths7262 are inhibited. In an embodiment, an inhibiting element 7261 is afield lens. In an embodiment, an inhibiting element 7261 is adiffractive waveguide. In an embodiment, an inhibiting element 7261 is acurved waveguide surface.

FIG. 7I illustrates an embodiment of the plurality of energy locations7118, wherein an energy inhibiting element 7264 provides an absorptiveproperty to limit the propagation of energy 7266 while allowing otherpropagation paths 7267 to pass.

FIG. 7J illustrates an embodiment of the plurality of energy locations7118, and the plurality of energy waveguides 7112, wherein a firstenergy inhibiting structure 7268 is configured to substantially orientenergy 7270 propagating therethrough into a first state. A second energyinhibiting structure 7271 is configured to allow energy 7270, which issubstantially oriented in the first state, to propagate therethrough,and to limit propagation of energy 7269 oriented substantiallydissimilarly to the first state. In order to further control energypropagation through a system, exemplified by the stray energypropagation 7272, energy inhibiting structures 7268, 7271 may require acompound energy inhibiting element to ensure energy propagationmaintains accurate propagation paths.

FIG. 7K illustrates an embodiment of the plurality of energy locations7118, and wherein an energy inhibiting element 7276 provides anabsorptive property to limit the propagation of energy along energypropagation path 7278 while allowing other energy along energypropagation path 7277 to pass through a pair of energy waveguides 7112for an effective aperture 7284 within the array of waveguides 7112. Inan embodiment, energy inhibiting element 7276 comprises black chrome. Inan embodiment, energy inhibiting element 7276 comprises an absorptivematerial. In an embodiment, energy inhibiting element 7276 comprises atransparent pixel array. In an embodiment, energy inhibiting element7276 comprises an anodized material.

FIG. 7L illustrates an embodiment comprising a plurality of energylocations 7118, and a plurality of energy waveguides 7112, wherein afirst energy inhibiting structure 7251 is placed adjacent to the surfaceof the energy locations 7118 and comprises a specified refractive,diffractive, reflective, or other energy altering property. The energyinhibiting structure 7251 may be configured to limit the first subset ofenergy propagation paths 7290 to a smaller range of propagation paths7275 by inhibiting propagation of energy along energy propagation paths7274. A second energy inhibiting structure 7261 is structured to alterenergy propagation paths 7275 to a certain extent depending upon whichof the plurality of energy locations 7118 the energy propagation paths7275 extends through. Energy inhibiting structure 7261 may alter energypropagation paths 7275 in a uniform or non-uniform way such that certainenergy propagation paths 7274 are inhibited. A third energy inhibitingstructure 7254 is placed orthogonal between regions of energy locations7118. The energy inhibiting structure 7254 exhibits an absorptiveproperty, and has a defined height along an energy propagation path 7275such that certain energy propagation paths 7274 are inhibited. An energyinhibiting element 7276 provides an absorptive property to limit thepropagation of energy 280 while allowing energy 7281 to pass through. Acompound system of similar or dissimilar waveguide elements 7112 arepositioned to substantially fill an effective waveguide element aperture7285 with energy from the plurality of energy locations 7118 and toalter the propagation path 7273 of energy as defined by a particularsystem.

In an embodiment, the energy inhibiting structure 7124 may be locatedproximate the first energy location 7122 and generally extend towardsthe first energy waveguide 7104. In an embodiment, the energy inhibitingstructure 7124 may be located proximate the first energy waveguide 7104and generally extend towards the first energy location 7122.

In one embodiment, the energy system is configured to direct energyalong the second plurality of energy propagation paths through theenergy waveguide system to the singular seamless energy surface, and todirect energy along the first plurality of energy propagation paths fromthe singular seamless energy surface through the energy relay system tothe plurality of energy devices.

In another embodiment, the energy system is configured to direct energyalong the first plurality of energy propagation paths from the pluralityof energy devices through the energy relay system to the singularseamless energy surface, and to direct energy along the second pluralityof energy propagation paths from the singular seamless energy surfacethrough the energy waveguide system.

In yet another embodiment, the singular seamless energy surface isoperable to guide localized light transmission to within three or lesswavelengths of visible light.

Sensory Data Suitable for Holographic Displays

The plenopic 4D function through the surface from an energy directingsurface provides for two spatial coordinates x_(l),y_(l) from a firstplane comprising energy locations and directed through a secondcoordinate along a second plane comprising waveguiding parametersu_(l),v_(l) defining a vector of an energy propagation pathƒ_(l)(x_(l),y_(l),u_(l),v_(l)). In consideration of a plurality ofenergy directing surfaces, the plenoptic 5D function provides for threespatial coordinates x_(l),y_(l),z_(l) from a first coordinate comprisingone or more energy locations and directed through a second coordinatealong a plane comprising waveguiding parameters u_(l),v₁ defining avector of an energy propagation path ƒ₁(x_(l),y_(l),z_(l),u_(l),v_(l)).For each of 4D or 5D, additional variables for time and colorƒ_(l)(λ_(l),t_(l)) may be considered and assumed to be inclusive of anyof the plenoptic functions as necessary for an application even when notexplicitly noted for simplicity of the function and discussion. For theavoidance of doubt, the reference to an energy directing surface is forexemplary purposes only and may comprise any additional point, location,direction, or plane in space for the localization of a 5D coordinate,and collectively referred to as an energy “directing surface”.

FIG. 8 is a flow chart diagram illustrating an embodiment of a process800 for determining four dimensional (4D) plenoptic coordinates forcontent data. The process 800 may include a step 802 in which contentdata is received, which may include any signal perceptible by a visual,audio, textural, sensational, or olfactory sensor. FIG. 9 is a schematicdiagram illustrating an embodiment of the content data, which mayinclude at least one of the following: an object location, a materialproperty (such as material properties 906, 907, and 908), a virtuallight source 904, geometry 902 at non-object location, content out ofthe reference surface, a virtual camera position 914, a segmentation 910of objects, background texture 912, and layered contents.

Referring to FIGS. 8 and 9 , the process 800 may further include a step804 in which locations of data points are determined with respect to afirst surface 920 to create a digital volumetric representation 922 ofthe content data. The first surface 920 may be used as a referencesurface for defining the locations of data points in space. In anembodiment, the process 800 may further include a step 806 in which 4Dplenoptic coordinates of the data points are determined at a secondsurface by tracing the locations of the data points in the volumetricrepresentation to the second surface where a 4D function is applied. Inan embodiment, the process 800 may further include a step 808 in whichenergy source location values are determined for 4D plenopticcoordinates that have a first point of convergence.

The content data received in step 802 may include N views, where N isone or more. A single view may be presented with or without a depthchannel. Stereoscopic views may be presented with or without a depthchannel. Multi-view imagery may be presented with or without a depthchannel. Further, a 4D light field may be presented with or without adepth channel.

The tracing of step 806 may use prior knowledge of a calibrated geometryof an energy system, which may be stored in memory as a global model oran individually characterized system or some combination of the twomethodologies.

In an embodiment, the mapping between the input data and the outputenergy source provides a methodology to accurately map between variousbitrate sources. The tracing of step 806 provides the ability to inferthe full volumetric 4D data set from the above listed partial samples.Depth information either needs to be provided or calculated from theavailable data. With the depth information known or calculated, the Nview(s) may be inverse traced by triangulation of the samples from theknown volumetric presentation based upon depth coordinate into the 4Dspace.

The triangulation may assume that each available energy source locationin the N source content are representative of a energy source locationfor each energy waveguide in the event that a mapping between energywaveguide and energy source location format resolution are provided. Inthe event that the N source content resolution are lower,super-resolution or scaling algorithms may be implemented. In the eventthat the resolution of the N source image(s) are higher than the numberof energy waveguides in the energy directing device, interpolationbetween super-sampled energy source locations may be performed toproduce higher amount of energy source locations per energy waveguide inthe resultant 4D inverse ray trace.

The above assumes distance information may be determined from the depthmaps which may or may not be accurate depending on the form of depthinformation provided or calculated, and with the distance informationknown or assumed, the distance information in combination with the x-yenergy source location coordinate and the (u,v) angular information asdetermined by the energy directing device properties may then beconsidered a 4D or 5D light field with limited imaging data samples. Theimaging samples, based upon the distance information, are triangulatedback to the appropriate energy source locations that may exist behindeach energy waveguide respectively, and missing data may be generated instep 808 through the disclosures contained herein.

Referring to FIGS. 7C, 8, 9, 10 in an embodiment, the energy locationsmay be located in the first surface 920, and the second surface where a4D function is applied may correspond to a waveguide system 7100 of anenergy directing device, and energy is operable to be directed throughthe waveguide system according to the 4D plenoptic coordinates of thedata points to form a detectable volumetric representation of thecontent data.

In an embodiment, the process 800 may further comprise a step 810, inwhich energy source location values are determined for 4D coordinatesthat have a first point of convergence. To provide an exampleimplementation of the present disclosure, FIG. 10 illustrates anembodiment of an energy directing device 700 going through a tracingprocess where content data in the form of an image 702 is provided witha distance position 704, which may be provided or calculated, within adetermined minimum position 706 and maximum position 708 in reference tothe energy locations 710. In an embodiment, the energy locations 710 maycomprise an energy directing device surface. The known geometry from theenergy locations 710 defined by the 4D plenoptic function allows for thetriangulation of a point 714 on the virtual surface of the image 702 tobe traced back along rays 716 to specific energy locations 710, eachhaving a unique x-y coordinate. Missing samples may be computationallycalculated based upon the available information contained within thedataset.

When additional N samples are provided, the same methodology is appliedwith the additional multi-perspective imaging data producing a richerset of inverse ray traced samples and provide superior holographicresults. The depth information from a multiple N samples may be providedthrough a single depth map, or up to N, or greater than N depth mapswith a known mapping between the source location (the N+X perspective)and the source depth map (the N+X depth map) to ensure appropriateinverse ray tracing is performed.

In the event that a singular depth map for the, for example, center Nperspective is provided, the additional depth maps may be interpolatedby calculating for the disparity between each of the adjacent views toaccurately map the source and target location between the N and the N+Xviewpoints. With this method, it is possible to inverse ray trace theappropriate view dependent mapping to the 4D light field such that thecorrect perspective(s) are projected to the appropriate waveguidecoordinates and results in the viewer's ability to maintain the correctview dependencies in the associated viewpoints.

The encoder and decoders are robust and may interpret multiple datatypes to include, but not limited to, 2D/flat files, 2D with depth,stereoscopic, stereoscopic with single depth channel, stereoscopic withdual depth channel, N+X multi-view with no depth, N+X multi-view withN+Y depth, geometric or vector based scene files that may includetextures, geometry, lighting, material properties and the like toreconstruct an environment, deep imaging files wherein multiple RGBAZvalues may be provided for each x-y coordinate, 4D or 5D (4D plus depth)light fields, or provided as a N+X view plus N+Y delta channel datasetwherein the depth channel provides a lower bandwidth methodology foronly rendering a certain amount of energy source location data asrequired for a determined energy directing device field of view. Theprocessors are able to inverse ray trace at up to, or exceeding,real-time speeds, in order to provision the appropriate 4D light fieldto present to the viewer with and without world coordinate locations,with and without compensated minimum and maximum projected worldlocations and in consideration of the energy directing device intrinsicas characterized and/or designed.

In an embodiment, the process 800 may further comprise a step 812, inwhich a mapping between energy locations 7122 on a first side of thewaveguide system 7100 and the angular directions of the energypropagation paths 7120 from the waveguide element 7100 on a second sideof the waveguide system 7100 is applied. Doing so may allow a pluralityof energy locations on the first side of the waveguide system 7100corresponding to the 4D plenoptic coordinates of the data points to bedetermined.

FIG. 12 is a schematic diagram of a processing system 1200 comprising adata input/output interface 1201 in communication with a processingsubsystem having a sensory data processor 1202, a vectorization engine1204, and a tracing engine 1206. It is to be appreciated that thesensory data processor 1202, the vectorization engine 1204, and thetracing engine 1206 may be implement on one or more processors, whetherindividually or any combination thereof. Step 802 of the process 800 mayinput content data through the data input/output interface 1201 to theprocessing subsystem 1220. Step 804 may be performed by the sensory dataprocessor 1202 to create a volumetric representation of the contentdata. Step 806

In an embodiment, applying the mapping may comprise calibrating for adistortion in the waveguide system 7100, which may further comprisecalibrating for at least one distortion selected from a group consistingof: a spatial distortion, angular distortion, intensity distortion, andcolor distortion.

In an embodiment, the energy directing device may further comprise arelay system 6110 on the first side of the waveguide system 7100, therelay system having a first surface 6116 adjacent to the waveguidesystem 7100, and the energy locations 7112 on the first side of thewaveguide system may be positioned adjacent to a second surface 6114 ofthe relay system 6110.

In an embodiment, applying the mapping may include calibrating for adistortion in the waveguide system 7100. In an embodiment, applying themapping may include calibrating both for a distortion in the relaysystem 6110 and a distortion in the waveguide system 7100. In anembodiment. the distortion to be calibrated may include at least onedistortion selected from a group consisting of: a spatial distortion,angular distortion, intensity distortion, and color distortion.

In an embodiment, a portion of the method may be carried out in realtime, or the method may be entirely carried out in real time, or atleast two portions of the method may be carried out in different timeperiods.

2D to Light Field Conversion

In an embodiment, content data may comprise data points in a twodimensional (2D) space, and determining locations of step 704 maycomprise applying a depth map to the data points in a two dimensionalspace.

There are several methods to convert two-dimensional or flat imageryinto light field data. These include the estimation of depth informationthrough depth from motion analysis, a provided depth channel throughmanual or rendered means, or the manual creation of disparity, depth,occlusion, geometry and/or any other methodology known as standard forvisual effects content creation to reproduce the full light fieldthrough regeneration of the entire environment through manual andautomated processes.

In a first embodiment, a system that includes a real-time or offlineprocessor to perform estimation of depth from available energy sourcelocation information is possible. This may be performed at the energydirecting device, as a set top box or as an offline process. Additionalcomputation for missing volumetric data may be performed leveragingtemporal information and/or state of the art texture synthesis or othertechnologies known in the art.

In a second embodiment, depth information is provided as an image streamand may be embedded into the image format. Similarly, additionalcomputation may be performed for missing volumetric data.

In a third embodiment, an artist or a process is leveraged to generatethe missing environmental information which may include a process toisolate or segment each object in a scene, track said objects over timemanually, semi-automatically or automatically, place objects into spaceleveraging disparity space, energy directing device space, optical spaceor world coordinates, synthesizing background and foreground missinginformation through visual effects processes known in the art forreconstruction of backgrounds, transparencies, edge details, etc. toregenerate the environment. For the avoidance of doubt, the implementedprocesses may be any, none or all of the listed embodiments for thereconstruction of these environments. The generated environmentalinformation should include as much of the missing information aspossible as determined by the energy directing device angles of view,and these angles of view may be known by the artist to ensure thatappropriate occlusion and view dependent information is generatedappropriately.

Additionally, the surface model for each object in the scene may begenerated, either as a partial model or as a completely built model andtextures from the image data are projected onto the surfaces of thegeometry to provide appropriate shape for the following inverse raytracing.

Additionally, material properties may be calculated or manuallyintroduced to ensure that view dependent lighting may be introduced withvirtual illumination sources to further increase the accuracy of theregeneration of the 4D light field.

Further, the addition of CG or synthetic content may be introduced toaugment the existing converted materials. The addition of volumetricdata may also be incorporated. The inter-mixing of N+X content may beintroduced as well to provide a seamless blend between CG, 2D,stereoscopic, multiview and/or 4D media into a single composite.

The resultant 2D to light field converted content may be retained as ageometric scene file including geometry, textures, lighting, materials,etc. as indicated in the CG scene itself, rendered as N+X views with N+Ddepth channels, rendered as a 4D or 5D (4D+depth) light field, a deepimage which is a format that allows for multiple RGBAZ samples per x-yenergy source location coordinate with or without a limitation ofstacking of Z samples per x-y coordinate, or provided as a N+X view plusN+Y delta channel dataset wherein the depth channel provides a lowerbandwidth methodology for only rendering a certain amount of energysource location data as required for a determined energy directingdevice field of view. Tools may be provided to allow for the generationof all, some or one of these respective output formats.

Stereoscopic and Multi-View to Light Field Conversion

The process from above leveraging single view content may be applied tostereoscopic and multi-view materials. The estimation of depthinformation is obtained through depth from motion analysis, as well asfrom stereoscopic, multi-view and/or disparity analysis, a provideddepth channel or provided depth channels through manual or renderedmeans, or the manual creation of disparity, depth, occlusion, geometryand/or any other methodology known as standard for visual effectscontent creation to reproduce the full light field through regenerationof the entire environment through manual and automated processes andleveraging the appropriate data to further retain the view dependentcontent as available in the provided imaging materials.

In an embodiment, the content data received in step 102 may comprisedata points in a three dimensional (3D) space, and determining locationsmay comprise adjusting the data points in the 3D space.

In an embodiment, adjusting the data points in the 3D space may includeapplying a depth map to the data points in the 3D space, adding new datapoints, reconstructing occluded data points, or any combination thereof.

The significant advantage to this approach exists in that the accuracyof stereoscopic disparity estimation is far greater than from motionparallax or other similar 2D estimation processes alone. Further theimage quality of the resultant converted 4D light field is more accuratedue to the availability of some of the view dependent conditions,including but not limited to illumination, transparencies, materials,occlusion, etc.

The ability to retain the explicit angular dependencies of themulti-view image data relies on the ability to calculate the surfacenormals in relation to the center viewpoint camera, or some otherdefined center point. With these normals and disparity or depthinformation known, it is possible to interpolate between viewpointsbased upon energy directing device angle of view, which is then eitherdirectly applied to the inverse ray tracing, or synthesized as a portionof the texture synthesis during the inverse ray tracing.

For brevity, all of the previously disclosed methodologies for thereconstruction of 2D to light field imagery may be applied to thereconstruction of stereoscopic or multi-view datasets.

Generation of N×N RGB Images from 4D or 5D Light Fields

By leveraging 4D or 5D light fields, it is possible to generate N×N orany value of up to N×N number of RGB multi-view images. This process isaccommodated by considering each bottom left coordinate under eachwaveguide, assuming a square grid, the 0,0 position, and the top rightposition as the N,N position. The grid is only exemplary and any othermapping methodology may be leveraged. For each 0,0 to N,N position, itis possible to form full resolution images from the light field with thewidest possible depth of field based upon the capture system leveragedwherein each waveguide in the array is considered a single energy sourcelocation and each coordinate under each waveguide is a single energysource location of the larger energy source location array for eachcomplete image from 0,0 to N,N respectively. This may be repeated for a5D light field for the depth information as well. In this fashion, it ispossible to easily translate between the 4D or 5D light field to anysubset of the dataset that is desired for various distribution reasonsto include 2D, stereoscopic, multi-view, point cloud, CG scene file, orany other desired combination of data that may be derived from the 4D or5D light field. For non-regular or square packed 4D or 5D structures,further interpolation is required to align energy source locations to aregular grid, or a linear mapping between energy source locations andnon-square packed structures may be implemented wherein the resultantimages may not appear rectilinear and may also contain energy sourcelocation artifacts.

FIG. 11 exemplifies the methodology to convert from a 4D or 5D lightfield into multiple viewpoints by arranging the energy locations 1102from underneath of each energy waveguide element 1104 according toenergy waveguide element position and energy location coordinaterespectively. This provides the ability to seamlessly transfer betweenlight field and smaller datasets seamlessly.

N+X RGB and N+Y Depth Datasets

The ideal dataset format that provides the highest quality with thebalance of data transmission size includes the use of N+X RGB and N+YDepth+vectorized channels wherein N+X RGB information contains N RGBimages that may represent a certain resolution and format, and X thatmay represent a different resolution and format for RGB data to includelower resolutions, delta information and the like, and N+YDepth+vectorized channels that contains N depth+vectorized channels thatmay represent a certain resolution and format and Y that may represent adifferent resolution and format for depth+vector data to include lowerresolutions, delta information and the like.

The number of N+X views may be generated on a regular grid, from aradius around a center point with or without a center view, frommultiple radii around a center point with or without a center view, orany methodology to determine the mapping of the number of views and theassociated packing or perspective locations. The configuration for theperspectives may be contained in the metadata of the file, or thedepth+vectorized channels provided may include a direct mapping to worldcoordinates such that the imaging data aligns to the same coordinate inXYZ space without other necessary metadata.

4D Disk Inversion and Energy Directing Device Compatibility Processing

For any data captured with a plenoptic or light field 4D or 5D system,including potentially those captured with virtual rigs with opticalsimulation of a 4D or 5D light field system, the resultant fly's eyeperspectives contain discs that represent the uv vectors for the lightfield. However, these coordinates assume energy focusing elements thatmay not exist in an energy directing device. In the proposed energydirecting device solution, the focusing elements may be the viewer'seye, and the mapping between the capture system and the mapping betweenthe original capture methodology and the viewed energy directing deviceare no longer correct.

To invert this and correct for the additionally missing energy directingelement in the system when compared to the capture system, it ispossible to individually flip each disc independently, wherein the x-ylocation of each (u,v) coordinate is retargeted based upon the centerpoint of each waveguide respectively. In this fashion, the inversion ofthe image that forms as a result of the main waveguide is inverted andallows for the light field energy directing device to project the raysin the correct x-y-u-v orientation.

A further embodiment of this may implement a hardware modificationwherein leveraging an energy waveguide array provides a direct inversionof every presented energy waveguide energy source location. For lightfield energy directing devices, this is advantageous to have a directmapping between a potential capture system and energy directing device.This may further be advantageous an embodiment comprising HMD systems orvolumetric opacity energy directing devices such that a group of energywaveguides in the overall array may be eliminated by removing thenecessity to relay additional times for accurate x-y-u-v coordinates.

Further, not all light fields are identical. They may be captured withdiffering NAs, FOVs, N values, optical prescriptions, etc. Theintrinsics and extrinsics of the input light field data may beunderstood and convert to the energy directing device characteristics.This may be performed by embodiments contained within this disclosurefor universal parametization of holographic and light field data.

Universal Parameterization of Holographic Sensory Data Transport ThroughInverse EnergyTracing and Vectorization of Sensory Properties for anEnergy Directing System

The plenopic 4D function through the surface from an energy directingsurface provides for two spatial coordinates x_(l),y_(l) from a firstplane comprising energy locations and directed through a secondcoordinate along a second plane comprising waveguiding parametersu_(l),v_(l) defining a vector of an energy propagation pathƒ_(l)(x_(l),y_(l),u_(l),v_(l)). In consideration of a plurality ofenergy directing surfaces, the plenoptic 5D function provides for threespatial coordinates x_(l),y_(l),z_(l) from a first coordinate comprisingone or more energy locations and directed through a second coordinatealong a plane comprising waveguiding parameters u_(l),v_(l) defining avector of an energy propagation pathƒ_(l)(x_(l),y_(l),z_(l),u_(l),v_(l)). For each of 4D or 5D, additionalvariables for time and color ƒ_(l)(λ_(l),t_(l)) may be considered andassumed to be inclusive of any of the plenoptic functions as necessaryfor an application even when not explicitly noted for simplicity of thefunction and discussion. For the avoidance of doubt, the reference to anenergy directing surface is for exemplary purposes only and may compriseany additional point, location, direction, or plane in space for thelocalization of a 5D coordinate, and collectively referred to as anenergy “directing surface”.

Along a first vector of an energy propagation path, a plurality ofintersection points comprising convergence of energies may occurtogether with additional energy propagation paths. At this intersectionpoint, a 3D point or depth parameter forms at location X_(l),Y_(l),Z_(l)among the plurality of energy propagation paths with the 4D or 5Dfunctions, wherein the 3D point of convergence X_(l),Y_(l),Z_(l) amongthe plurality of energy propagation paths, where for each x_(l),y_(l) orx_(l),y_(l),z_(l) coordinate contained within the energy directingsurface or surfaces, there is only a single u_(l),v_(l) propagation paththat forms between a first coordinate and the converging 3D point. The4D function ƒ_(Z)(x_(l),y_(l),u_(l),v_(l)) or 5D functionƒ_(Z)(x_(l),y_(l),z_(l),u_(l),v_(l)) collectively define all 4Dx_(l),y_(l), or 5D x_(l),y_(l),z_(l) coordinates and commensurateu_(l),v_(l) propagation paths that exist for each converging point atX_(l),Y_(l),Z_(l).

At a first 5D coordinate resulting from the convergence of energiesalong a plurality of energy propagation paths through the energydirecting surface X_(l),Y_(l),Z_(l), the coordinate may represent apoint within a larger object, volume, particle or localized energyparameter, wherein converging energies at additional coordinatesproximate to the first 5D coordinate may exhibit additional vectorizedproperties for sensory energies within an environment or holographicdataset. These vectorized properties may comprise information for each5D coordinate, for each energy location coordinate within the 4Ddataset, for regions within either of the 4D or 5D datasets, or othersub-sets of coordinates comprising the energy surface.

In an embodiment, the universal parameterization of 4D and 5Dholographic sensory energy properties for propagation of visual,auditory, somatosensory, gustatory, olfactory, vestibular or otherdesired energies for sensory system response for raster and vector 2D,3D, 4D and 5D datasets are disclosed, wherein the 2D data may comprise asingle angular sample, 3D data may comprise two or more angular samplesin a single dimension, 4D data may comprise a plurality of angularsamples in two dimensions, or 5D data may comprise a plurality ofangular samples in three or more dimensions, in reference to the secondcoordinate of the second plane of the 4D energy directing surface.

Embodiments of received sample data may comprise any of:

-   -   1). 2D or monoscopic, flat, point cloud, uv-mapped geometry,        intrinsic geometry, deep images, layered images, CAD files        (intrinsic), single point sampling, single camera capture,        single projector projection, volumetric (monoscopic single        sample points with vectors in a volume), sources of 3 Degrees of        Freedom (DoF; raster with monoscopic x, y, z rotation about a        single point), sources of non-light field 6 DoF (raster+vectors        from monoscopic samples), volumetric energy directing device        (monoscopic samples in a volume), sources of Pepper's Ghost        (single point projection), sources of 2D AR HMD (monoscopic        single or multiple focus planes; layered monoscopic), sources of        2D VR HMD (monoscopic single or multiple focus planes; layered        monoscopic), or any other representation of two-dimensional        raster or vector information;    -   2). 3D or stereoscopic, triscopic (single baseline), multiview        (1D), 1D multi-sample, 1D multi-perspective, horizontal or        vertical only parallax, 1D projection array, two point sampling,        1D point sampling, horizontal or vertical array, bullet time,        sources of 3 DoF (raster; stereoscopic x, y, z rotation about a        single point), sources of 3 DoF (3D raster within stereoscopic        x, y, z rotation about a single point), sources of non-light        field 6 DoF (3D raster+vectors from stereoscopic samples),        sources of 1D volumetric energy directing device (1D parallax        contained samples), sources of autostereoscopic data, sources of        horizontal multiview energy directing device, sources of 3D AR        HMD (stereoscopic single or multiple focus plane; layered        stereoscopic), sources of 3D VR HMD (stereoscopic single or        multiple focus planes; layered stereoscopic), or any other        representation of three-dimensional raster or vector        information;    -   3). 4D or plenoptic (5D), multiscopic, integral image, light        field (4D), holographic (4D), 2D multiview, 2D multi-sample, 2D        multi-perspective, 2D parallax, horizontal and vertical        parallax, 2D projection array, 2D point sampling, motion capture        stage (along a surface), planar array, witness camera array,        rendered or raytraced geometric representations (4D        representations), extrinsic geometry (4D representation),        sources of light field 6 DoF (4D raster within planar light        field samples), sources of free-viewpoint 6 DoF (4D        raster+vectors from 4D light field samples), sources of 4D        volumetric energy directing device (2D parallax contained        samples), sources of light field energy directing device (4D        sampling), sources of light field HMD (near field 4D sampling),        sources of holographic energy directing device (4D sampling), or        any other representation of four-dimensional raster or vector        information;    -   4). 5D or plenoptic+depth, light field+depth, holographic (5D        sampling, 4D+depth), arbitrary multiview (along all x, y and z        axis), multi-sample (along all xyz), multi-perspective (along        all xyz), volumetric parallax (along all xyz), projection array        (along all xyz), point sampling (along all xyz), motion capture        stage (along all xyz), witness camera array (arbitrary xyz        configurations), rendered or raytraced geometric representations        (5D representations), cubic or volumetric rendering (along all        xyz), extrinsic geometry (5D representation), sources of light        field 6 DoF (5D raster within volumetric light field samples),        sources of free-viewpoint 6 DoF (5D raster+vectors from 5D light        field samples), sources of 5D volumetric energy directing device        (multiplanar 4D sampling), sources of 5D light field energy        directing device (5D sampling, 4D+multiple planes), sources of        5D light field HMD (near field 5D sampling, 4D+multiple planes),        sources of holographic energy directing device (5D sampling,        4D+multiple planes), or any other representation of        five-dimensional raster or vector information.

At each of the second coordinates, the provided data may comprise asub-set or a super-set of either raster or vector samples and whereinsamples may represent and include additional vectorized information toenable transformation into increased sampling density throughinterpretation or processing of the sub-set or super-set of raster orvector samples.

For each of 2D, 3D, 4D or 5D provided datasets, the information isconverted through vectorized information, manual identification,computer vision analysis, automated processing, or other means totransform the provided samples from the original dataset into a 5Dcoordinate system. For each of 2D, 3D, 4D or 5D provided datasets, theinformation may comprise multiple samples or layers of samples as wellas additional vectorized properties in respect to the originatingangular sampling component for each provided dataset in reference to thesecond coordinate of the second plane of the 4D energy directingsurface, or may comprise a combination of contributing samples for anyof 2D, 3D, 4D or 5D additional provided datasets.

Each of the provided samples comprise intrinsic energy for each desiredcoordinate, wherein the intrinsic energy may include additionalextrinsic energy attributes, where the intrinsic energy represents valueat a given 5D coordinate in the absence of other external samples,properties or environmental conditions. In the electromagnetic spectrum,this may be referred to as the albedo as the dimensionless measurementfor reflectance corresponding to a white body that reflects all incidentradiation, but explicitly extended to each desired sensory energywherein the range of dimensionless values is commensurate to thespecified sensory energy. Within the visual sensory systems, this rangeis approximately 400 nm to 700 um, and in the auditory sensory systems,this range is approximately 20 Hz to 20 kHz.

Over the past several decades, vast technological improvements enablingthe reproduction of human senses artificially leveraging sophisticatedpattern recognition of detected sensation, aromas and flavors throughelectronic means. For other systems that may exist outside of theelectromagnetic spectrum, these dimensionless values may becharacterized in the same way based upon sensed acuity response. Whileholographic sensory energy technologies are newly emerging, disclosedwithin this embodiment comprises a system, method and format for thestimulation of all human senses in a virtual environment to articulatethe universal construct for various sensory parameters wherebyprovisioning for the appropriate data handling, transmission, storage,vectorization, translation to, from and between any sensory energyparameter or device desired for complete immersion of the constructedvirtual environment and embodiments of energy propagation forholographic sensory technologies will be disclosed in futureapplications. It is additionally the intent of this disclosure to enableother analogue devices, including novelties like the classic“smell-o-vision,” or contemporary versions like FeelReal's smelling VRheadset, to leverage the parameterized values provided for within thevectorization of the dataset herein.

In an embodiment, the somatosensory system may be defined based upon thecomponents that define sensitivity including mechanoreceptors fortextures with a pressure sensitivity range in the skin that may benormalized between 50 Hz to 300 Hz, thermoreceptors with a temperaturesensitivity range in the skin that may be normalized between 0° c. to50° c. (although this range may be much wider range with upper and lowerbounds defined by the extremes of temperature) or surface deformabilitydefining the range of viscoelastic behaviors of a material measure bothviscous and elastic characteristics when undergoing deformations betweenstress and strain over time which provides for a multiplicity of physicsincluding variables for time, strain, modulus, among other dynamics, andfor the purposes of this disclosure is simplified to a dimensionlessnormalized scale with a value of 0 for unmovable solids such as granite,and 1 for low viscosity liquids such as water. Those skilled in the artwill understand that the actual vectors provided will comprise thenecessary physics to appropriately define the viscoelasticity of thematerial, and normalized for exemplary purposes only.

Finally, state of the art advances in artificial electronic sensingincluding gustatory and olfactory devices demonstrate a viable path tofurther vectorizing the sensory parameters disclosed for the Holodeckdesign parameters, as well as enable the electronic reproduction ofartificial taste and smell through a holographic waveguiding means asdescribed herein. Artificial electronic taste and smell receptors havemade considerable progress through emerging nanodevices, whereinfrequency-based artificial taste receptors using an enzymatic biosensorto sample the intensity of chemical stimulus through the encoding andconversion to frequency based pulses to both repeatedly and accuratelydetect taste as frequencies of the sampled chemical compositions througha pattern recognition system resulting in the detection of the tastesthat compose the human palate. It is believed that the technology may beextended to all types of detectable tastes and similar advances inartificial olfactory system have demonstrated digital interfaces forstimulating ones smell receptors using weak electrical pulses targetingthe nasal conchae with ongoing studies to further parameterize thepatterns contained within frequencies of particular olfactory responsesthrough variation in electrical signals.

With the path established for the arbitrary generation of frequenciesand complex electronic patterns to represent olfactory, gustatory andother sensory system, in one embodiment, the acuity response for tastemay be vectorized to comprise a normalized scale for each ofelectronically controlled parameters along a scale from 0 to 1 torepresent the minimum and maximum gustatory response to saturate theaverage human's 2,000-8,000 taste buds, potentially comprising but notlimited to vectors for sourness, saltiness, bitter (spiciness),sweetness, and savoriness (unmami) wherein the vector and the spatialcoordinate of the vectorized signals may inform the production for thecomplex olfactory implementations.

In another embodiment, the acuity response for smell may be furthervectorized to comprise a normalized scale for each of electronicallycontrolled parameters along a scale from 0 to 1 to represent the minimumand maximum olfactory response to saturate the average human's 10 cm′ ofolfactory epithelium, for each of the highly complex olfactory spacespotentially comprising but not limited to vectors for fragrant, fruity,citrus, woody (resinous), chemical, sweet, mint (peppermint), toasted(nutty), pungent and decayed wherein the vector and the spatialcoordinate of the vectorized signals may inform the production for thecomplex olfactory implementations.

Each of these vectors may provide the normalized values representingthese patterns for taste, smell or other sensory domains, converted to awave, amplitude, magnitude or other attribute as required for theappropriate application of the provided vectorized values. While thesense of smell and taste are two of the most highly debased senseswithin the sensory system, with parameterized values to vectorizecomplex amalgamations, it is additionally possible in an embodiment toprovide for user based interactive control over the sensitivity of anysuch sensory energy to provide for customization of individualization ofeach of visual, auditory, somatosensory, gustatory, olfactory,vestibular or other desired sensory system responses.

In an embodiment, each of the represented sensory albedo energy valuesof the sample may additionally comprise extrinsic energy attributesbaked into the single sample value representing the additive result ofeach provided sample respective of other external samples, properties orenvironmental conditions. In this configuration, the compound samplevalue may or may not exhibit latent attributes of other energies fromother samples in a physically based or simulated environment. The mostefficient and pure methodology to transmit the parameterized andreconstructed holographic dataset is based upon the singular intrinsicsample information providing for simplified and lower bandwidthfrequency information, although this is not always possible to receiveoutside of entirely synthetic environments, particularly for physicallybased imaging or acoustic systems. In any real-world environment, thereis always some amount of extrinsic contribution to the resultant sampleinformation. Certain systems like the Light Stage, or other systemsknown in the art to facilitate the estimation of reflectance, shape,texture, and motion capture leverage some form of structuredillumination and one or more imaging devices which provide for thedirect or indirect analysis of the albedo, depth information, surfacenormal and bidirectional scattering distribution surface properties.

The bidirectional scattering distribution function (BSDF) is ageneralized superset of the bidirectional transmittance distributionfunction (BTDF), the bidirectional texture function (BTF), and thebidirectional reflectance distribution function (BRDF), which are oftenrepresented by the generalized function ƒ_(r)(w_(i),w_(r)), collectivelyact as a model to parameterize and identify surface properties incomputer graphics and vision algorithms known in the art. The functiondescribes how visible light is reflected, transmitted or otherwiseinteracts with a surface given an incoming incident direction w_(i) andoutgoing reflected or transmitted direction w_(r) for an energypropagation path, where the surface normal is perpendicular to thetangent of the object surface and the function describes the ratio ofreflected radiance exiting along the outgoing path w_(r) to theirradiance incident on the surface along the incoming path w_(i),wherein each of w_(i),w_(r) may comprise a 4D function to define aparameterized azimuth and zenith angle for each of the incoming lightpath and the exiting light path.

The functions may further be articulated for a first location x_(i) ofenergy λ_(i) striking a surface, and exit after material propertiesinternally scatter the energy to a second location x_(r) of energy λ_(r)to account for visible wavelength effects like iridescence,luminescence, subsurface scattering, non-local scattering effects,specularity, shadowing, masking, interreflections, or the like,resultant output energy based upon material properties of a surface, theinput energies and locations, the output energies and locations acrossthe surface of an object, volume, or point.

Therefore, the generalized properties to describe how energy istransported between any two energy rays that strike a surface, toinclude wavelength or frequency dependency and spatially varyingmaterial properties or surfaces, may be represented as a 10D function,and specified as ƒ_(r) (λ_(i),x_(i),w_(i),λ_(r),x_(r),w_(r)) for each orany of the available or provided samples within a dataset to account forinput energy, the impact of a vectorized surface profile, and the outputreflected, refracted, specular, transmitted, scattered, diffused, orother material property result from any energy domain given thegeneralization of the function ƒ_(r).

In consideration now of the energy directing surface, the plenopic 4Dfunction provides for two spatial coordinates x_(l),y_(l) from a firstplane comprising energy locations and directed through a secondcoordinate along a second plane comprising waveguiding parametersu_(i),v_(i) defining a vector of an energy propagation pathƒ_(l)(x_(l),y_(l),u_(l),v_(i)). In consideration of a plurality ofenergy directing surfaces, the plenoptic 5D function provides for threespatial coordinates x_(l),y_(l),z_(l) from a first coordinate comprisingone or more energy locations and directed through a second coordinatealong a plane comprising waveguiding parameters u_(i),v_(i) defining avector of an energy propagation pathƒ_(i)(x_(l),y_(l),z_(l),u_(l),v_(l)). For each of 4D or 5D, additionalvariables for time and color ƒ_(l)(λ_(l),t_(l)) may be considered andassumed to be inclusive of any of the plenoptic functions as necessaryfor an application even when not explicitly noted for simplicity of thefunction and discussion.

Along a first vector of an energy propagation path, a plurality ofintersection points comprising convergence of energies may occurtogether with additional energy propagation paths. At this intersectionpoint, a 3D point or depth parameter forms at location X_(l),Y_(l),Z_(l)among the plurality of energy propagation paths with the 4D or 5Dplenoptic functions, wherein the 3D point of convergenceX_(l),Y_(l),Z_(l) among the plurality of energy propagation paths, wherefor each x_(l),y_(l) or x_(l),y_(l),z_(l) coordinate contained withinthe energy directing 4D surface or 5D surfaces, there is only a singleu_(l),v_(l) propagation path angle that forms between a first coordinateand the converging 3D point. The 4D functionƒ_(Z)(x_(l),y_(l),u_(l),v_(l)) or 5D function ƒ_(Z)(x_(l),y_(l),z_(l),u_(l),v_(l)) collectively define all 4D x_(l),y_(l),or 5D x_(l),y_(l),z_(l) coordinates and commensurate u_(l),v_(l)propagation paths that exist for each converging point atX_(l),Y_(l),Z_(l).

At a converging coordinate X_(l),Y_(l),Z_(l), a surface is formed andthe surface may comprise a point, volume, object or other embodimentcomprising a 3D position of converging energy propagation paths. Theprovided samples for each surface location may comprise one or moresurface properties, vectors, materials, characterizations, or otheridentifying property V_(i) to characterize or otherwise process theresulting energy, as well as one or more input energy sources striking agiven point proximate to the surface location wherein the reflectancefunction now comprises a generalized vector for the various propertiesof the surface and represented as an 11D universal objectparameterization function ƒ_(r) (λ_(i),x_(i),w_(i),λ_(r),x_(r),w_(r),V_(i)).

The 11D universal holographic parameterization functionƒ_(r)(λ_(i),x_(i),w_(i),λ_(r),x_(r),w_(r), V_(i)) defines the resultantvalues for a given environment and vectorized object properties and the4D function ƒ_(l)(x_(l),y_(l),u_(l),v_(l)) defines the energypropagation paths from an energy directing device surface, may thereforebe further generalized as an 15D universal holographic parameterizationfunction ƒ_(r)(λ_(i),x_(i),w_(i),λ_(r),x_(r),w_(r)(x_(l),y_(l),u_(l),v_(l)), V_(i))where the transmitted direction w_(r) defines and equals the propagationpath of u_(l),v_(l), whereby defining the spatial coordinate x_(l),y_(l)and for each transmitted direction w_(r) there may be only oneƒ_(l)(x_(l),y_(l),u_(l),v_(l)) set of values to satisfyw_(r)=u_(l),v_(l). Those skilled in the art will appreciate the varioustransforms and mathematical constructs in addition to the renderingrequirements associated with the disclosed universal parameterization of4D and 5D holographic sensory energy properties.

With the complete 15D function describing the vectorization of allsensory energy properties to coincide with surfaces formed fromconverging points in space, multiple orders of magnitude of requireddata have been fundamentally eliminated provisioning for a viable pathto enabling the transmission of truly holographic datasets.

The vectorized properties strive to provide accurate physics for each ofthe sensory domains for properties that may be synthetically programmed,captured, or computationally assessed, wherein V_(l) may prescribeattributes for each surface, volume or 3D coordinate X_(l),Y_(l),Z_(l)vectorized properties about an object for a given sample within aprovided dataset for general system metadata or for each or any sensoryenergy domain, comprising:

-   -   1.) system metadata may provide for any of the sensory energy        specific attributes or system wide references for surface        properties for each sample including normals, depth information,        environmental properties, multiple angular samples for a given        3D coordinate, procedural textures, geometry, point clouds, deep        image data, static frames, temporal frames, video data, surface        IDs, surface passes, coordinate maps, virtual camera        coordinates, virtual illumination and visible energy        information, environment maps, scene information outside of the        field of the visual sensory sample information, curves,        vertices, temporal information, networked data, databases,        object recognition, energy devices, external data feeds, sensors        for system modifications and interactivity, system status, voice        recognition, olfactory detection, auditory detection, facial        recognition, somatosensory recognition, gustatory recognition,        UI, UX, user profiles, flow and motion vectors, layers, regions,        transparency, segments, animation, sequence information,        procedural information, displacement maps, or any other scene        data that is necessary to provide sufficient data for the        appropriate processing of each sample;    -   2.) visual sensory energy may provide surface properties to        define the appropriate rendering of visible or non-visible        electromagnetic energy, iridescence, luminescence, subsurface        scattering, non-local scattering effects, specularity,        shadowing, absorbance, transmission, masking, interreflections,        albedo, transparency, physics, dynamics, reflection, refraction,        diffraction, optical effects, atmospheric effects, frequency,        modulation, surface profiles, textures, displacement maps,        physics and dynamics to specifically interrelate to other        sensory energies and respond based upon provisioned energies        (e.g. vibrations of sound altering reflectance properties or        tactile material deformation causing surface deformations),        layers, regions, transparency, segments, curves, animation,        sequence information, procedural information, size of material,        environmental conditions, room dynamics, or other related        material properties for a surface, environment, room, object,        point, volume or the like;    -   3.) auditory sensory energy: vectors related to the placement of        localized sound fields, magnitude, amplitude, mass, material        propagation parameters, absorbance, transmission, material        properties informing acoustic reflectance, diffusion,        transmission, augmentation, masking, scattering, localization,        frequency dependence or modulation, pitch, tone, viscosity,        smoothness, texture, modulus, any other parameters that        determine the propagation of acoustic waves within the object,        surface, medium or otherwise, physics and dynamics to        specifically interrelated to other sensory energies and respond        based upon provisioned energies (e.g. temperature changing the        sound of a material), layers, regions, transparency, segments,        curves, animation, sequence information, procedural information,        size of material, environmental conditions, room dynamics, or        other related material properties for a surface, environment,        room, object, point, volume or the like;    -   4.) somatosensory energy vectors related to the mechanoreceptors        for textures, pressure, thermoreceptors, temperature, surface        deformability parameters and vectors defining the range of        viscoelastic behaviors of a material measure both viscous and        elastic characteristics when undergoing deformations between        stress and strain over time which provides for a multiplicity of        physics including variables for time, strain, modulus, among        other dynamics, layers, regions, transparency, segments, curves,        animation, sequence information, procedural information, size of        material, environmental conditions, room dynamics, or other        related material properties for a surface, environment, room,        object, point, volume or other somatosensory parameters;    -   5.) gustatory sensory energy vectors for fragrant, fruity,        citrus, woody (resinous), chemical, sweet, mint (peppermint),        toasted (nutty), pungent and decayed wherein the vector and the        spatial coordinate of the vectorized signals may inform the        production for the complex olfactory implementations and further        provide duration, magnitude, frequency, length, time, radius,        modulation, layers, regions, transparency, segments, curves,        animation, sequence information, procedural information, size of        material, environmental conditions, room dynamics, or other        related material properties for a surface, environment, room,        object, point, volume or other gustatory sensory parameters;    -   6.) olfactory sensory energy vectors for sourness, saltiness,        bitter (spiciness), sweetness, and savoriness (unmami) wherein        the vector and the spatial coordinate of the vectorized signals        may inform the production for the complex olfactory        implementations and further provide duration, magnitude,        frequency, length, time, radius, modulation, layers, regions,        transparency, segments, curves, animation, sequence information,        procedural information, size of material, environmental        conditions, room dynamics, or other related material properties        for a surface, environment, room, object, point, volume or other        olfactory parameters;    -   7.) or other interrelated sensory dynamics based upon physical,        synthetic, transmitted, or computational interdependencies from        any other sensory sample dataset, sensory system vectors as        needed, designed, or required and any additional sensory        properties where parameterization of a particular characteristic        is beneficial for the reconstruction, storage, processing or        transmission of generalized holographic constructed data.

With the received dataset comprising 2D data having a single angularsample, 3D data having two or more angular samples in a singledimension, 4D data having a plurality of angular samples in twodimensions, or 5D data having a plurality of angular samples in three ormore dimensions.

For all provided source materials, each source material may undergoadditional processes to appropriately prepare for efficientvectorization of the holographic dataset. For any provided sourcematerials that exhibit lower spatial or angular resolution that theenergy directing surface, a transformation process may be required inorder to accurately convert the originating source to a 4D or 5Ddataset.

For appropriate preparation, in an embodiment, provided 2D or 3D sourcematerials comprise photographic capture from a standard imaging system.Within this sequence of images are rastered reflections, refractions,transparent elements and other similar examples of material propertyinteraction with physically based illumination.

In the event that the content is prepared by simply identifying surfaceIDs for the surfaces with the already rastered material properties, theeffective data may be sufficient for converging into a 4D coordinatesystem, however, any additional rendering applied to these surfaces willexhibit a double image for the physics of both the photographic, as wellas the parameterized synthetic rendered reflectance properties. Theideal source dataset for efficient holographic transmission comprises analbedo representation of the sample source information, plus vectorizedmaterial properties for each of the specified energy domains withmetadata forming an object-based volumetric sampling of the albedomulti-view samples, and wherein all material properties provide foraccurate surface identification and rendering as well as thelocalization or projection of other sensory energies accurately basedupon the specified vectorized surface properties.

In an embodiment, manual, semi-automated, computer vision, or automatedprocesses are provisioned to algorithmically or manually assess thecontent within the source sample dataset, and wherein a manual oralgorithmic analysis is performed whereby segmentation and other objectisolation methodologies known in the art are performed to identify theregions that include undesired physically rasterized effects. In anembodiment, a person is photographed in front of a background whereinthe material properties of the person include reflections from theenvironment, and the background objects are occluded by the photographedperson. After these regions have been identified as undesirable, aprocess may be leveraged to 1) isolate the objects in question; 2)separate all object elements into the core components to account forocclusion, transparency, edges, or other element; 3) through imageanalysis, temporal analysis, energy analysis, with the facilitation ofmachine learning, computer vision, extra hardware and energy devicesthat additionally captured information about the scene, objects and/orenvironment, or through completely manual means, the object elements areprovisioned such that any surface that should exhibit a materialproperty has any such baked-in material properties removed throughcomputer vision, algorithms, processors, or manual visual effectswherein the manual processes are generally known in the art for methodsto perform wire removals, paint fix, clean plates, image restoration,alpha matte creation, occlusion filling, object recreation, imageprojection, motion tracking, camera tracking, rotoscope, optical flow,and the like for the purpose of regenerating the intrinsic materialproperty in the absence of the extrinsic material properties therebypreparing the content for the most efficient transmission andpropagation for said dataset; 4) An additional process of the aboveinvolves the manual or computer assisted identification of depth or 3Dcoordinate values for each of the desired samples; and 5) Further withinthis embodiment is the identification of the associated materialproperties, each of which represent a point, region of data, surface,object or other representation of a material such that the data mayeasily be further rendered within the energy directing device's displaydrivers or within any additional system capable of either encoding anddecoding the parameterized dataset.

In an embodiment, the dataset from the above comprises 3D multiviewsamples that are prepared with albedo visual energy samples, each ofwhich having multiple layers of rgba information, a collection ofvectorized material properties to associate each segmented material witha surface ID and series of surface parameters to closely reconstruct theoriginal source dataset prior to the removal of the extrinsic imagedata, and wherein an acoustic dataset is provisioned with vectorizedmaterial properties associated with the material properties of thevisual energy system as well as multiple sound channels each havingidentified frequency, modulation, spatial placement and other soundlocalization properties, and wherein a somatosensory sensory energydataset is provided for a subset of the surfaces contained within thevisual energy dataset, to additionally comprise viscoelastic andtemperature vectorized material properties, both of which are correlatedto the other vectorized datasets.

From any provided dataset, each provided sample from the visual energydataset is assessed for a relative depth position in relation to theenergy directing device surface, and wherein each of the samples for anyof the visual energy samples are placed into a 3D coordinate system, andwherein the energy propagation path length for each of the providedsamples is assessed in relation to the function that correlates each 3Dcoordinate in relation to the plurality of coexisting converging energypropagation paths that intersection a first 3D point at locationX_(l),Y_(l),Z_(l) among the plurality of energy propagation paths withinthe 4D or 5D plenoptic functions, where for each x_(l),y_(l) orx_(l),y_(l),z_(l) coordinate contained within the energy directing 4Dsurface or 5D surfaces, there is only a single u_(l),v_(l) propagationpath angle that forms between a first coordinate and the converging 3Dpoint. The 4D function ƒ_(Z) (x_(l),y_(l),u_(l),v_(l)) or 5D functionƒ_(Z) (x_(l),y_(l),z_(l),u_(l),v_(l)) collectively define all 4Dx_(l),y_(l), or 5D x_(l),y_(l),z_(l) coordinates contained within theenergy directing device and commensurate u_(l),v_(l) propagation pathsthat exist for each converging point at X_(l),Y_(l),Z_(l) and whereinthe total number of samples per presented or available 4D x_(l),y_(l),or 5D x_(l),y_(l),z_(l) spatial coordinates is known after performingthis analysis process, and wherein the total energy propagation pathlength between each 3D point at location X_(l),Y_(l),Z_(l) to the 4D or5D coordinate location is known, and wherein a weighted distributionbased upon total available samples per 4D or 5D coordinate and minimumpath length to the sampled 3D coordinate values from the availableplurality of 3D coordinate data provides for a complete sampling of the4D or 5D light field from an arbitrary dataset.

As a further embodiment of the above, after each of the samples for anyof the 1) visual, acoustic, somatosensory, and any other provided energysamples are 2) placed into a 3D coordinate system based upon theprovided dataset, additional processing, or additional vectorizedproperties, and before performing a coordinate analysis; 3) the 15Duniversal holographic parameterization function ƒ_(r)(λ_(i),x_(i),w_(i),λ_(r),x_(r),w_(r) (x_(l),y_(l),u_(l),v_(l)) isassessed wherein 4) additional known environmental scene, geometry,metadata or the like is provided, each with independent vectorizedmaterial properties; 5) virtual illumination information is provided andthe additional sensory energy metadata properties are assessed for anypotential interference between the properties that may altering therendering functions and; 6) the 15D parameterization function assessesfor each provided 3D coordinate and commensurate vectorized materialproperty to; 7) perform a rendering process through on-line, off-line,real-time, processor, ASIC, FPGA, cloud, or other form of renderingprocess to result in a new plurality of angularly varying materialproperties given the arbitrary provided dataset, and wherein 8) therendering process is specific to each of the transmitted direction w_(r)defining and equal to each of the propagation paths u_(l),v_(l), wherebydefining the spatial coordinate x_(l),y_(l), and for each transmitteddirection w_(r) there may be only one ƒ_(l)(x_(l),y_(l),u_(l),v_(l)) setof values to satisfy w_(r)=u_(l),v_(l), and wherein 9) based upon therendered results and resultant available new angularly varying materialproperties, for each of the 4D or 5D coordinates comprising the energypropagation path length for each of the provided samples are assessed inrelation to the function that correlates each 3D coordinate in relationto the plurality of coexisting converging energy propagation paths thatintersection a first 3D point at location X_(l),Y_(l),Z_(l) among theplurality of energy propagation paths within the 4D or 5D plenopticfunctions, where for each x_(l),y_(l) or x_(l),y_(l),z_(l) coordinatecontained within the energy directing 4D surface or 5D surfaces, thereis only a single u_(l),v_(l) propagation path angle that forms between afirst coordinate and the converging 3D point. The 4D functionƒ_(Z)(x_(l),y_(l),u_(l),v_(l)) or 5D functionƒ_(Z)(x_(l),y_(l),z_(l),u_(l),v_(l)) collectively define all 4Dx_(l),y_(l), or 5D x_(l),y_(l),z_(l) coordinates contained within theenergy directing device and commensurate u_(l),v_(l) propagation pathsthat exist for each converging point at X_(l),Y_(l),Z_(l) and whereinthe total number of samples per presented or available 4D x_(l),y_(l),or 5D x_(l),y_(l),z_(l) spatial coordinates is known after performingthis analysis process, and wherein the total energy propagation pathlength between each 3D point at location X_(l),Y_(l),Z_(l) to the 4D or5D coordinate location is known, and wherein a weighted distributionbased upon total available samples per 4D or 5D coordinate and minimumpath length to the sampled 3D coordinate values from the availableplurality of 3D coordinate data provides for a complete sampling of the4D or 5D light field for all provided sensory energies from an arbitrarydataset.

An additional embodiment of the above system wherein the renderingadditionally accounts for a bidirectional energy directing surface suchthat sensed electromagnetic energy representing the illumination of thereal-world environment, or the absorbance of certain acousticfrequencies within the environment may result in the dynamic or off-lineupdate to the rendering process or other sensed interactive real-worldelement is assessed, and wherein the illumination and acoustic or othersources are adjusted to accommodate for the modification inenvironmental conditions.

Turning back to FIG. 8 , in view of the principles disclosed above, inan embodiment of process 800, the received content data may furthercomprise vectorized material property data, and wherein the process 800further comprises a step 830, in which digital volumetric representationof the content data is associated with the vectorized material propertydata; and wherein, in step 804, determining energy source locationvalues is based on at least the vectorized material property dataassociated with the volumetric representation of the content data.

Referring to FIGS. 9 and 13 , in an embodiment, a vectorization process1300 may include a step 1302 in which first content data is received anda step 1304 in which identifying a surface 915 in the content data. Inan embodiment, identifying the surface 915 may comprise usingsegmentation data in the content data. The vectorization process 1300may further include a step 1306 in which a surface identification of thesurface 915 is determined and a step 1308 in which material propertydata of the surface 915 is determined. In an embodiment, determining thematerial property data may comprise manual determination, or using apredetermined process. After steps 1306 and 1308, the vectorizationprocess 1300 may further include a step 1310 in which the surfaceidentification is associated with the material property data of thesurface 915. The vectorization process 1300 may further include a stepof 1312 in which the vectors of the material property data is created.The vectorization process 1300 may further include a step 1314 in whichvectorized material property data is generated based on the createdvectors.

In an embodiment, the process 1300 may optionally include a step 1316 inwhich material property data is removed from the first content data andreplaced by the vectorized material property data is generated in step1314. In an embodiment the vectorized material property data isgenerated in step 1314 may used in process 800 as discussed above todetermine 4D plenoptic coordinates for the energy directing devices ofthe present disclosure as discussed above.

The process 1300 may be carried out using any processing system of thepresent disclosure, including processing system 1200. In an embodiment,content data may be received in step 1302 through the data input/outputinterface 1201, and steps 1304 thru 1314 of the vectorization process1300 may be carried out using the vectorization engine 1204.Additionally, the vectorized material property data generated in step1314 may be used by the sensory data processor 1202 and tracing engine1206 for processing according to the steps of process 800 as discussedabove. Steps 808 and 812 may be performed by the tracing engine todetermine 4D coordinates for holographic presentation. Step 810 may beperformed by the sensory data processor 1202. The output of theprocessing subsystem may be provided to a compression engine 1210, fromwhich compressed data may be stored in a memory or provided to the datainput out interface 1201 for transmission to an energy directing systemeither connected locally or remotely to the system 1210. Data may alsobe stored in the memory 1208 until a later time to be retrieved.

A hogel is a part of a light-field hologram. Figure Hogels may berendered

While various embodiments in accordance with the principles disclosedherein have been described above, it should be understood that they havebeen presented by way of example only, and are not limiting. Thus, thebreadth and scope of the invention(s) should not be limited by any ofthe above-described exemplary embodiments, but should be defined only inaccordance with the claims and their equivalents issuing from thisdisclosure. Furthermore, the above advantages and features are providedin described embodiments, but shall not limit the application of suchissued claims to processes and structures accomplishing any or all ofthe above advantages.

It will be understood that the principal features of this disclosure canbe employed in various embodiments without departing from the scope ofthe disclosure. Those skilled in the art will recognize, or be able toascertain using no more than routine experimentation, numerousequivalents to the specific procedures described herein. Suchequivalents are considered to be within the scope of this disclosure andare covered by the claims.

Additionally, the section headings herein are provided for consistencywith the suggestions under 37 CFR 1.77 or otherwise to provideorganizational cues. These headings shall not limit or characterize theinvention(s) set out in any claims that may issue from this disclosure.Specifically, and by way of example, although the headings refer to a“Field of Invention,” such claims should not be limited by the languageunder this heading to describe the so-called technical field. Further, adescription of technology in the “Background of the Invention” sectionis not to be construed as an admission that technology is prior art toany invention(s) in this disclosure. Neither is the “Summary” to beconsidered a characterization of the invention(s) set forth in issuedclaims. Furthermore, any reference in this disclosure to “invention” inthe singular should not be used to argue that there is only a singlepoint of novelty in this disclosure. Multiple inventions may be setforth according to the limitations of the multiple claims issuing fromthis disclosure, and such claims accordingly define the invention(s),and their equivalents, that are protected thereby. In all instances, thescope of such claims shall be considered on their own merits in light ofthis disclosure, but should not be constrained by the headings set forthherein.

The use of the word “a” or “an” when used in conjunction with the term“comprising” in the claims and/or the specification may mean “one,” butit is also consistent with the meaning of “one or more,” “at least one,”and “one or more than one.” The use of the term “or” in the claims isused to mean “and/or” unless explicitly indicated to refer toalternatives only or the alternatives are mutually exclusive, althoughthe disclosure supports a definition that refers to only alternativesand “and/or.” Throughout this application, the term “about” is used toindicate that a value includes the inherent variation of error for thedevice, the method being employed to determine the value, or thevariation that exists among the study subjects. In general, but subjectto the preceding discussion, a numerical value herein that is modifiedby a word of approximation such as “about” may vary from the statedvalue by at least ±1, 2, 3, 4, 5, 6, 7, 10, 12 or 15%.

As used in this specification and claim(s), the words “comprising” (andany form of comprising, such as “comprise” and “comprises”), “having”(and any form of having, such as “have” and “has”), “including” (and anyform of including, such as “includes” and “include”) or “containing”(and any form of containing, such as “contains” and “contain”) areinclusive or open-ended and do not exclude additional, unrecitedelements or method steps.

Words of comparison, measurement, and timing such as “at the time,”“equivalent,” “during,” “complete,” and the like should be understood tomean “substantially at the time,” “substantially equivalent,”“substantially during,” “substantially complete,” etc., where“substantially” means that such comparisons, measurements, and timingsare practicable to accomplish the implicitly or expressly stated desiredresult. Words relating to relative position of elements such as “near,”“proximate to,” and “adjacent to” shall mean sufficiently close to havea material effect upon the respective system element interactions. Otherwords of approximation similarly refer to a condition that when somodified is understood to not necessarily be absolute or perfect butwould be considered close enough to those of ordinary skill in the artto warrant designating the condition as being present. The extent towhich the description may vary will depend on how great a change can beinstituted and still have one of ordinary skilled in the art recognizethe modified feature as still having the required characteristics andcapabilities of the unmodified feature.

The term “or combinations thereof” as used herein refers to allpermutations and combinations of the listed items preceding the term.For example, “A, B, C, or combinations thereof is intended to include atleast one of: A, B, C, AB, AC, BC, or ABC, and if order is important ina particular context, also BA, CA, CB, CBA, BCA, ACB, BAC, or CAB.Continuing with this example, expressly included are combinations thatcontain repeats of one or more item or term, such as BB, AAA, AB, BBC,AAABCCCC, CBBAAA, CABABB, and so forth. The skilled artisan willunderstand that typically there is no limit on the number of items orterms in any combination, unless otherwise apparent from the context.

All of the compositions and/or methods disclosed and claimed herein canbe made and executed without undue experimentation in light of thepresent disclosure. While the compositions and methods of thisdisclosure have been described in terms of preferred embodiments, itwill be apparent to those of skill in the art that variations may beapplied to the compositions and/or methods and in the steps or in thesequence of steps of the method described herein without departing fromthe concept, spirit and scope of the disclosure. All such similarsubstitutes and modifications apparent to those skilled in the art aredeemed to be within the spirit, scope and concept of the disclosure asdefined by the appended claims.

What is claimed is:
 1. A method of directing energy based on fourdimensional (4D) coordinates for content data, the method comprising:receiving content data; determining a location for each of a pluralityof data points with respect to a first surface to create a digitalvolumetric representation of the content data, the first surface being areference surface for defining the locations of data points in space;tracing the location of each of the data points to the first surfacethrough a plurality of locations at a second surface where energy is tobe propagated therethrough in accordance to a 4D function applied at thesecond surface; determining a set of 4D coordinates for each data pointbased on a plurality of traced intersections at the first and secondsurfaces; determining a first set of energy source modulation values fora first set of 4D coordinates based on the 4D function, the first set of4D coordinates associated with a first point of convergence at alocation of a first data point; and determining a second set of energysource modulation values for a second set of 4D coordinates based on the4D function, the second set of 4D coordinates associated with a secondpoint of convergence at a location of a second data point that isdifferent than the location of the first data point; wherein the firstand second sets of 4D coordinates are determined based on differentpluralities of traced intersections at the first and second surfaces;and wherein the first set of energy source modulation values is operableto instruct an energy directing device to direct energy to the firstpoint of convergence along a first plurality of energy propagation pathsto form at least a first portion of a detectable volumetricrepresentation of the content data, and the second set of energy sourcemodulation values is operable to instruct the energy directing device todirect energy to the second point of convergence along a secondplurality of energy propagation paths to form at least a second portionof the detectable volumetric representation of the content data; whereinthe first and second portions of the detectable volumetricrepresentation of the content data comprise a holographic surface thatis perceivable to a viewer.
 2. The method of claim 1, wherein thecontent data comprises a signal perceptible by a visual, audio,textural, sensational, or smell senor.
 3. The method of claim 1, whereinthe content data comprises at least one of the following information ofa scene: an object location, a material property, a virtual lightsource, content for geometry at non-object location, content out of thereference surface, a virtual camera position, a segmentation of objects,and layered contents.
 4. The method of claim 1, wherein the content datacomprises data points in a two dimensional (2D) space, and whereindetermining locations comprises applying a depth map to the data pointsin a two dimensional space.
 5. The method of claim 1, wherein thecontent data comprises data points in a three dimensional (3D) space,and wherein determine locations comprises adjusting the data points inthe 3D space.
 6. The method of claim 5, wherein adjusting comprisesapplying a depth map to the data points in the 3D space.
 7. The methodof claim 5, wherein adjusting comprises adding new data points.
 8. Themethod of claim 5, wherein adjusting comprises reconstructing occludeddata points.
 9. The method of claim 1, wherein the second surfacecorresponds to a waveguide system of an energy directing device, andenergy is operable to be directed through the waveguide system accordingto the 4D plenoptic coordinates of the data points to form a detectablevolumetric representation of the content data.
 10. The method of claim9, wherein the method further comprises applying a mapping betweenenergy locations on a first side of the waveguide system and the angulardirections of the energy propagation paths from the waveguide element ona second side of the waveguide system, wherein a plurality of energylocations on the first side of the waveguide system corresponding to the4D plenoptic coordinates of the data points are determined by applyingthe mapping.
 11. The method of claim 10, wherein applying the mappingcomprises calibrating for a distortion in the waveguide system.
 12. Themethod of claim 11, calibrating for the distortion in the waveguidesystem comprises calibrating for at least one distortion selected from agroup consisting of: a spatial distortion, angular distortion, intensitydistortion, and color distortion.
 13. The method of claim 9, wherein theenergy directing device further comprises a relay system on the firstside of the waveguide system, the relay system having a first surfaceadjacent to the waveguide system, and further wherein the energylocations on the first side of the waveguide system are positionedadjacent to a second surface of the relay system.
 14. The method ofclaim 13, wherein applying the mapping comprises calibrating for adistortion in the waveguide system.
 15. The method of claim 13, whereinapplying the mapping comprises calibrating for a distortion in the relaysystem.
 16. The method of claim 15, wherein applying the mappingcomprises calibrating for a distortion in the waveguide system.
 17. Themethod of claim 15, wherein calibrating for the distortion in the relaysystem comprises calibrating for at least one distortion selected from agroup consisting of: a spatial distortion, angular distortion, intensitydistortion, and color distortion.
 18. The method of claim 9, wherein theenergy locations are located in the first surface.
 19. The method ofclaim 1, wherein the received content data further comprises vectorizedmaterial property data, and wherein the method further comprisesassociating the digital volumetric representation of the content datawith the vectorized material property data; and wherein determiningenergy source location values is based on at least the vectorizedmaterial property data associated with the volumetric representation ofthe content data.
 20. The method of claim 1, wherein at least a portionof the method is carried out in real time.
 21. The method of claim 1,wherein method is entirely carried out in real time.
 22. The method ofclaim 1, wherein at least two portions of the method are carried out indifferent time periods.
 23. A method of determining four-dimensional(4D) plenoptic coordinates for content data, the method comprising:receiving content data; determining locations of data points withrespect to a first surface to create a digital volumetric representationof the content data, the first surface being a reference surface;determining 4D plenoptic coordinates of the data points at a secondsurface by tracing the locations the data points in the volumetricrepresentation to the second surface where a 4D function is applied; anddetermining energy source location values for 4D plenoptic coordinatesthat have a first point of convergence; wherein the second surfacecorresponds to a waveguide system of an energy directing device, andenergy is operable to be directed through the waveguide system accordingto the 4D plenoptic coordinates of the data points to form a detectablevolumetric representation of the content data.